Alignment Is Local: A Paired Diagnostic for GUI Agents under User-Side Persuasion

arXiv:2607.29199 · cs.CR · Submitted 2026-07-31 · Read on arXiv

Haoxin An, Yunpeng Song, Zihao Bai, Zhongmin Cai, Guojun Xiong, Chenhao Lin, Wentao Chen, Feng Wei, Chao Shen

cs.CR

Submitted: 2026-07-31

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Alignment — 20× in this paper · explained in 1 episode
• Benchmark — 15× in this paper · explained in 2 episodes
• Robustness — 11× in this paper · explained in 5 episodes
• Grounding — 7× in this paper · explained in 3 episodes
• Benchmarking — 6× in this paper · explained in 3 episodes
• LLM-as-a-Judge — 2× in this paper · explained in 3 episodes
• Adversarial Attacks — 1× in this paper · explained in 1 episode
• Adversarial Robustness — 1× in this paper · explained in 3 episodes
• Annotator Agreement — 1× in this paper · explained in 1 episode
• Consistency — 1× in this paper · explained in 3 episodes
• Cosine Similarity — 1× in this paper · explained in 3 episodes
• Fidelity — 1× in this paper · explained in 2 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Foundation Models — 1× in this paper · explained in 7 episodes
• Instruction Tuning — 1× in this paper · explained in 2 episodes
• Mode Collapse — 1× in this paper · explained in 2 episodes
• Objective Function — 1× in this paper · explained in 2 episodes
• Verification — 1× in this paper · explained in 2 episodes
• Visual Grounding — 1× in this paper · explained in 2 episodes

Sources

• Qwen2.5-VL Technical Report
• LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet
• GPT-4 Technical Report
• UI-TARS: Pioneering Automated GUI Interaction with Native Agents
• Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception
• Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
• MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environments
• Universal and Transferable Adversarial Attacks on Aligned Language Models

Related papers

• SoK: AI-Augmented Binary Reversing
• Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
• Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
• Efficient Fuzzy PSI under One-Sided Assumptions
• Sealing the Audit-Runtime Gap for LLM Skills
• Token Composition: A Graph Based on EVM Logs