MissClick: Execution-Aware Adversarial Attacks on Coordinate Generation in GUI Grounding Models

summary

The gist

Recent GUI visual grounding models generate screen coordinates as sequences of digit tokens that are parsed into numerical values and mapped to executable clicks, creating a security vulnerability

In short

MissClick exploits how GUI grounding models convert visual data into screen coordinates using their place-value system. It proposes two attacks: one for untargeted disruption by pushing clicks far outside the target area, and one for targeted hijacking by optimizing specific digit probabilities. Results show these methods significantly outperform existing defenses.

Key concepts

Coordinate Structure: GUI grounding models output coordinates as text sequences that are parsed using place values (like tens, hundreds). This structure means changing a single digit in a high-order position causes a much larger physical shift in the final executed click than changing a low-order digit.
Soft Coordinate Displacement (MissClick-U): This is an untargeted attack that creates an adversarial example by calculating 'soft coordinates' based on expected digit values weighted by their place values. By minimizing the distance between this soft click and the true target, it maximizes the chance of pushing the final click far away from its intended location.
Place-Weighted Target-Digit Loss (MissClick-T): For targeted attacks, this method optimizes discrete digit choices by using a loss function that weights errors based on place value. This forces the optimization to focus on high-order digits because an error at the hundreds place has a much greater impact on the final coordinate than an error at the ones place.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Federated Learning — explained in 16 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes
Sample Complexity — explained in 15 episodes
Generalization — explained in 13 episodes

This episode discusses

The paper

MissClick: Execution-Aware Adversarial Attacks on Coordinate Generation in GUI Grounding Models · Read on arXiv

National University of Defense Technology

Recent GUI visual grounding models generate screen coordinates as digit-token sequences that are parsed into numerical values and mapped to executable clicks. This generation-to-execution interface creates an attack surface that existing objectives over visual representations or coordinate-token sequences do not explicitly model. Although each coordinate digit is predicted as a token, its spatial effect after parsing depends on decimal position: changing a hundreds-place digit by one shifts the coordinate by 100 units, whereas the same change at the ones place shifts it by one. This mismatch motivates attack objectives that account for both numerical coordinate structure and click execution. Moreover, untargeted and targeted attacks require different objectives because they aim to move the click outside the correct region and into an attacker-specified region, respectively. We propose MissClick, an execution-aware white-box attack that aligns optimization with click-level success conditions. MissClick-U maximizes soft-coordinate displacement for untargeted disruption, while MissClick-T minimizes a place-weighted target-digit loss for targeted redirection. On OS-Atlas and UGround across desktop, web, and mobile platforms, MissClick-U achieves untargeted success rates of 75.07% and 72.93% (+16.62 and +30.72 pp), while MissClick-T achieves targeted success rates of 44.86% and 62.67% (+31.73 and +47.06 pp). Among the evaluated objectives, soft-coordinate displacement performs best for untargeted attacks, whereas place-weighted target-digit optimization performs best for targeted attacks, supporting goal-specific execution-aware objective design.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "MissClick: Execution-Aware Adversarial Attacks on Coordinate Generation in GUI Grounding Models".

Jane: Recent GUI visual grounding models generate screen coordinates as sequences of digit tokens that are parsed into numerical values and mapped to executable clicks,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we've covered a lot about "MissClick: Execution-Aware Adversarial Attacks on Coordinate Generation in GUI Grounding Models," and it seems clear that these models generate coordinates with a hidden structure that we can exploit by targeting how those numbers are formed. The authors propose MissClick-U for general disruption and MissClick-T for specific hijacking, showing how the place-value structure dictates the effectiveness of each attack objective.

Jane: It’s true, Tom; the paper highlights that simply looking at the coordinate string isn't enough because the actual click displacement depends entirely on those digit values and their decimal positions when parsed into numbers. The implications are that we have to move beyond treating these outputs as simple text and start considering their numerical construction when designing security for these systems.

Lu: I think the main contribution is formalizing this dependency between a high-order digit change and its resulting large coordinate shift, which provides the theoretical backbone for why these specific attack objectives are so effective in their respective settings. It’s a solid foundation for future work in adversarial analysis.

Meng: From a practical standpoint, it suggests that when we build systems reliant on visual grounding for actions, we need to ensure the underlying coordinate generation pipeline is as resistant to structural manipulation as possible, focusing on hardening those high-order positional inputs.

Lalam: I see the bigger picture here; this work pushes us to consider how AI interacts with the physical world through structured data and demands that our defenses evolve to understand that structure rather than just looking at surface-level outputs.

Tom: That’s a great way to put it, Lalam; so while we're excited about the technical details of MissClick, the real impact is forcing a necessary shift in how we think about securing these complex AI-driven interfaces moving forward.

Conclusion: Tom: So, we're wrapping up our deep dive into "MissClick: Execution-Aware Adversarial Attacks on Coordinate Generation in GUI Grounding Models," and I want to recap what this paper really boils down to for our listeners. It’s essentially about showing that how a computer translates a visual guess into an actual click is vulnerable because of the way coordinates are structured numerically.

Jane: Exactly, Tom; it explains that these grounding models produce text sequences that become numbers, and we can exploit the place-value system—like knowing a digit in the hundreds place matters much more than one in the ones place—to make clicks go where we want them.

Lu: I think what’s fascinating is how they formalized this dependency; they show a direct link between a high-order digit change and a massive shift in the executed coordinate. It opens up some really creative possibilities for how we can design systems that are sensitive to those structural weaknesses.

Meng: From an engineering standpoint, it tells me that if we're building applications that rely on these models for automation, we have to treat those coordinate outputs not just as arbitrary numbers but as a structured input where position truly matters.

Lalam: I see this work suggesting that the way AI processes and maps visual information is inherently tied to its numerical representation, which could fundamentally alter how we build trust in AI-driven interfaces across our entire culture.

Tom: It really puts a spotlight on the execution phase of an AI system, showing that we can attack the final output by manipulating the underlying math rather than just tricking the initial image recognition part.

Jane: And when you look at it from a simple perspective, it means that even if an AI is good at seeing what's on screen, we can still mess with its final action because of how those numbers are calculated internally.

Lu: The authors clearly set up two distinct attack paths, MissClick-U for general disruption and MissClick-T for specific hijacking, which shows a sophisticated understanding of the target model’s architecture.

Meng: I’m thinking about the practical impact on security: if this technique is understood, we can start building defenses that specifically check for these types of coordinate manipulation during runtime.

Lalam: It suggests a future where the security protocols for visual systems aren't just about input validation, but about understanding the entire mathematical journey from pixels to precise action.

Tom: Exactly! So, while we’ve covered how it works, this paper really hammers home that by attacking the place-value structure of coordinates, we can design defenses that are far more resilient against these types of manipulation.

Jane: And this whole concept is about making sure that the AI's understanding of its output isn't just a surface-level prediction but something we can actually control or at least predict within defined boundaries.

More episodes

← Home