Quantifying the Impact of Motion on 2D Gaze Estimation in Real-World Mobile Interactions

summary

The gist

The gist Mobile gaze tracking accuracy is significantly impacted by user mobility and behavior, with head distance, head pose, and device orientation being key factors affecting accuracy How it works

In short

The research investigated how user movement and behavior affect 2D gaze estimation using a front-facing camera on mobile devices. They found that head distance, head pose, and device orientation are critical factors influencing accuracy. Dynamic movements like walking or task switching significantly increase estimation errors compared to static use, highlighting the need for systems that explicitly model these changes.

Key concepts

Appearance-based 2D gaze estimation: This is the common method used in mobile eye tracking where a computer looks at a standard front-facing camera image to guess where the user is looking on the screen. It relies purely on visual cues from that single 2D image, making it accessible without needing specialized hardware.
Motion-related geometry change: This refers to how the physical relationship between your head, the camera, and your device changes constantly while you are using a phone. When you move or switch tasks (like lying down or walking), these spatial relationships shift, which confuses the simple 2D gaze estimation algorithm.
Head distance: This is a key measurement of how far your head is from the camera lens. The study found that this distance is most stable during continuous use but changes significantly when users switch between different activities or tasks, directly impacting how accurately the gaze is estimated.
Task-switch behavior: This describes when a user stops one activity and starts another, such as moving from sitting to walking or navigating a maze. The research showed that these switches cause much larger fluctuations in geometry and consequently lead to greater errors in gaze estimation than staying within the same task.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 18 episodes
Federated Learning — explained in 17 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes
Quantum Fisher Information (QFI) — explained in 15 episodes

This episode discusses

The paper

Quantifying the Impact of Motion on 2D Gaze Estimation in Real-World Mobile Interactions · Read on arXiv

Yaxiong Lei, Yuheng Wang, Fergus Buchanan, Mingyue Zhao, Yusuke Sugano, Shijing He, Mohamed Khamis

University of St Andrews University of Tokyo King’s College London University of Glasgow

Mobile gaze tracking on handheld devices commonly relies on appearance-based 2D gaze estimation from the device's front-facing RGB camera. Although this approach makes eye tracking more accessible by avoiding dedicated infrared hardware, maintaining accuracy remains difficult because the spatial relationship among the user's face, the camera, and the screen changes continuously during handheld use. This paper examines how motion-related geometry change affects a representative front-camera RGB 2D gaze-estimation pipeline. We report two complementary within-subject user studies (n=10) that capture handheld behaviour, calibration data, and gaze-estimation error during everyday-style mobile tasks across conditions ranging from lying and sitting to walking and maze navigation. Rather than treating motion as a single binary factor, we analyse within-task versus task-switch behaviour, calibration-test mismatch, and condition-specific motion signatures. Behavioural results show that head-to-camera distance is comparatively stable during sustained within-task use, while task switches introduce much larger geometry fluctuations, with head-distance standard deviations of approximately 7.42--10.10,cm across conditions. In the gaze-evaluation study, one-off calibration reduced average RMSE from 4.49,cm to 3.44,cm; under this regime, the global effect of motion condition on RMSE was not statistically reliable, whereas clearer patterns appeared in task-specific and motion-specific calibration analyses, where better calibration-test match reduced error.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Quantifying the Impact of Motion on 2D Gaze Estimation in Real-World Mobile Interactions".

Jane: The gist Mobile gaze tracking accuracy is significantly impacted by user mobility and behavior, with head distance, head pose,

Tom: First, who's behind it and why it matters.

Title and authors: Jane: Now, what does this paper actually summarize? It sets up a framework to capture both device movement and the visual information simultaneously. They use data from an accelerometer, gyroscope, magnetometer, camera frames, and ground-truth gaze to precisely capture both device movement and corresponding visual information >

Tom: So it’s a dual study design. First they characterize motion patterns in natural usage—User Study one—and then they assess how those patterns impact gaze estimation accuracy under simulated real-world conditions in User Study two > <ref:2502.10570#pg3>

Lu: User Study one focuses on observing and recording various motion states, like lying down or walking through a maze, to understand typical user interaction patterns > <ref:2502.10570#pg3>

Meng: And User Study two takes those same tasks but adds the actual gaze estimation procedures, like calibration and testing to measure accuracy under different motion conditions > <ref:2502.10570#pg1>

Jane: So they’re essentially mapping out how different physical activities translate into errors in the gaze tracking system.

The paper's summary: Tom: When you look at the raw data analysis from this paper, they find distinct behavioral patterns between static activities, like lying down, and dynamic ones, like walking >

Lu: They found that the largest variation in head distance happens when users are lying down—they recorded a mean of thirty-one point three seven centimeters with a standard deviation of ten point zero nine centimeters > <ref:2502.10570#pg2>

Jane: That’s quite a spread for distance, which is interesting because they also look at orientation, and lying down showed the largest variation there too >

Tom: The two dynamic conditions—walking slowly and walking in a maze—showed greater variation in accelerometer and gyroscope readings compared to the static conditions >

Meng: And they used t-SNE plots to show that lying down is distinctly separate from sitting, but walking slowly and walking in a maze are overlapping because their motion signatures are similar >

Jane: So they’re showing that the way you move really dictates how much the tracking system struggles.

The paper's improvements: Tom: When it comes to performance impact, they used a repeated-measures ANOVA and found a significant main effect of motion condition on gaze estimation error >

Lu: Specifically, dynamic conditions like walking and walking in a maze showed significantly higher errors than static conditions >

Jane: They also tested motion-specific calibration approaches, and those achieved very high accuracy with errors of less than two centimeters in stable conditions > <ref:2502.10570#pg1>

Tom: But under dynamic conditions, that error jumps up to two point seven three centimeters, which is a drop of nearly forty-eight point nine one percent compared to the one-off calibration approach >

Meng: The Lasso regression analysis pointed out that head distance was the most critical factor influencing accuracy across all motion conditions >

Lu: Head distance contributed nine point seven nine to the between-motion model, which is about thirty-five point eight seven percent of the total weight in that calculation >

Jane: So they’re suggesting that explicitly modeling head and device orientation is a big unmet need in 2D gaze estimation for mobile devices > <ref:2502.10570#pg1>

Conclusion: Tom: The main conclusion is that explicitly modeling head and device orientation remains the biggest missing piece in 2D gaze estimation for mobile devices > <ref:2502.10570#pg1>

Lu: They suggest that small posture shifts or task switches can degrade errors by up to two or three times >

Jane: So, the implication is that we need more advanced, adaptive eye-tracking systems that can explicitly account for head movements and maintain accuracy across diverse usage contexts >

Meng: If you're deploying this, you'd need something that adapts quickly to sudden changes in posture rather than relying on a one-time setup >

Lalam: I’m confident that incorporating orientation and head-distance data directly into the 2D pipeline is the most direct way to mitigate those performance degradations > <ref:2502.10570#pg1>

Tom: So, it sounds like the next step for this research is moving toward explicit modeling of these spatial factors rather than just relying on calibration adjustments >

More episodes

← Home