DEFLECT: Temporal Counterfactual Preference Learning for Delay-Robust Asynchronous VLAs

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "DEFLECT: Temporal Counterfactual Preference Learning for Delay-Robust Asynchronous VLAs".

Dev: Here's the gist of the paper. The gist The proposed DEFLECT framework converts latency-induced mismatch into counterfactual preference supervision to teach asynchronous Vision-Language-Action (VLA) policies to prefer execution-time actions over stale observations, thereby extending the usable delay envelope of asynchronous VLAs Vision-Language-Action (VLA) policies often use asynchronous inference to hide large-model latency behind ongoing robot motion, but this creates a prediction–execution mismatch where actions fit the prediction-time state but are misaligned with the execution-time state Existing strategies do not directly teach the policy to resolve this stale-input mismatch The key insight of DEFLECT is to use the unattainable zero-latency branch as an offline preference oracle for the deployable async branch, defining a temporal counterfactual preference where execution-time aligned chunks are preferred over stale ones How it works DEFLECT constructs temporal counterfactual supervision from offline expert trajectories by querying a frozen reference VLA twice under the same language instruction and shared sampling randomness, generating a preferred chunk from the future execution-time observation and a rejected chunk from the stale prediction-time observation The trainable policy then scores both chunks under the same deployment-time input used at test time, learning to favor execution-time-aligned actions while a supervised fine-tuning anchor preserves the expert action manifold This is achieved by minimizing an objective function that balances a Supervised Fine-Tuning (SFT) flow-matching term and a Flow-Matching Direct Preference Optimization (FM-DPO) term The final objective is LDEFLECT(θ) = λSFT LFM(θ; Aexp t, cdep t, τ, ϵ) expert anchor + λDPOLFM-DPO(θ) Preference Optimization under the Deployment-Time Input The preference pair is constructed as A+t = πref(st+dDPO, ot+dDPO), lt; ξ and A−t = πref(st, ot), lt; ξ, where dDPO is sampled during preprocessing to cover the latency range expected at deployment Both chunks are then scored under the same deployment-time input cdep t, which is the same input available at test time This ensures the policy is trained to answer the decision it will face at test time: given the stale deployment input, which chunk should be preferred The loss function LFM-DPO(θ) encourages the trainable policy to assign a lower relative flow loss to the preferred chunk A+t and a higher relative flow loss to the rejected chunk A−t under the deployment-time input Results and Generalization Across Kinetix, LIBERO, and three real-robot tasks, DEFLECT improves delay robustness over strong asynchronous VLA baselines, raising high-latency success by up to 6.4 percentage points in simulation and achieving a 4.6 percentage-point gain at the longest delay on a real-scale VLA In simulation, DEFLECT averages 73.5 over d ∈

5, 7: compared to VLASH’s 67.1, a substantial improvement of +6.4 on average and up to +7.0 at d=6 In real-world testing, DEFLECT improves on Conveyor-I success by +10 pp over VLASH (96.7 vs 86.7) and over π0.5 by +20 pp (96.7 vs 76.7) The analysis shows that the net DPO contribution is concentrated at higher delay, demonstrating that preference alignment concentrates its benefit precisely in the delay regime where fresh and stale observations produce different actions Mechanism and Conclusion DEFLECT is characterized as a dynamics-aware, variance-preserving correction because per-task variance changes are an order of magnitude larger than the mean displacements they accompany The training-input ablation study confirms that feeding both A+ and A− through the policy using the same input cdep t uniformly underperforms DEFLECT at every delay, showing that aligning the trained policy with the input it actually sees at inference is crucial DEFLECT does not change inference or add a scheduler, remaining a drop-in replacement for deployed VLASH checkpoints The framework suggests a general counterfactual-supervision recipe where both ideal and deployable conditions can be scored under the deployment context to provide a zero-cost preference signal Any method that resumes training from a pretrained checkpoint inherits a non-trivial lift from new optimizer state and LR warmup, with the net contribution of preference alignment being +1.3 at d=6 The final choice λDPO=0.02 is selected as the Pareto-optimal setting on the (delay-avg, horizon-avg) plane Limitations DEFLECT’s fresh/stale preference signal is weakest at low delays, where the two observations are nearly identical and the labeled contrast is small, so the DPO contribution mainly appears in the high-delay regime The paper notes that PFM underperforms VLASH at d ≥ 2, confirming that generic expert-vs-policy alignment makes things worse when applied to stale observations The additional compute cost is entirely offline, with reference forwards dominating the wall-clock overhead on Kinetix because they happen inside every gradient step Any gap to DEFLECT is shown to be attributable to the delay-aware preference signal, not just a tuning artifact of the smoothing parameter σ The results are robust across different training-delay distributions, with Model A (narrow) still recovering +3.7 at d=7 over VLASH The paper suggests that the preference signal is strongest when the scene or robot state changes substantially, as seen in the high-contrast regime The study concludes that DEFLECT improves high-delay execution by +6.4 percentage points over VLASH on Kinetix and a 4.6 percentage-point gain on LIBERO Implementation Details The preference-pair generation involves sampling A+ from the reference policy conditioned on the future observation (st+dDPO, ot+dDPO) and A− from the earlier observation (st, ot) The training involves AdamW with a cosine schedule, a 1,000-step linear warmup, and decay to 1% of peak LR The per-example flow-matching loss LFM(θ; A, c) is used as a surrogate for the negative log likelihood inside the DPO margin The training cost on Kinetix involves 11 reference forwards plus 1 trainable forward+backward per gradient step, compared to 1 trainable forward+backward for vanilla SFT In LIBERO, the cost is amortized to a one-time collection pass of roughly 3.7 × 10 6 reference forwards The paper also shows that removing the SFT anchor entirely causes a catastrophic collapse to 12.3 success rate The final choice λDPO=0.02 is the Pareto-optimal setting on the (delay-avg, horizon-avg) plane The performance is robust across a broad band λ ∈

0.01, 0.05: with delay-avg success rates between 83.0 and 83.3 The study confirms that the DPO margin takes the form of log-ratio margin that the original DPO derivation of Rafailov et al.

Title and authors: Rosa: So we're talking about this paper called DEFLECT, which deals with vision language action policies using asynchronous inference and how to make them more robust when there's a delay in getting new information. The authors are Yixiang Zhu, Yonghao Chen, Zijie Yang, Yusong Hu, and Xinyu Chen from the Hong Kong University of Science and Technology One Robotics.

Dev: Right, those are the folks who put this paper out on arXiv. The title itself says DEFLECT: Temporal Counterfactual Preference Learning for Delay-Robust Asynchronous VLAs.

Rosa: It’s about using temporal counterfactual preference learning to make these asynchronous VLA policies more robust against delays. Think of it as training the policy to choose better actions when the input it gets is stale compared to what actually happens during execution.

Taro: So, if I'm driving and I see a car in my rearview mirror, and I’m using an old prediction to decide whether to brake, DEFLECT tries to teach that policy how to handle that stale view better.

Dev: That’s a good way to put it. It moves away from just making the policy fast and instead focuses on making the policy smarter about the time difference between seeing something and acting on it.

The paper's summary: Rosa: So, what DEFLECT actually does is use offline expert trajectories—these are like perfect demonstrations from real robots—to create a kind of preference oracle. It queries a frozen reference VLA twice under the same language instruction and shared randomness to get two chunks of actions.

Dev: One chunk comes from the future execution-time observation, which is what actually happened, and the other comes from the stale prediction-time observation, which is what we usually see when we run things asynchronously.

Taro: So it’s creating a preference pair: one good action from where it should have gone, and one bad action from where it was predicted to go based on old data.

Rosa: Then the trainable policy scores both chunks using the same stale deployment input that we use at test time, learning to favor the execution-time-aligned chunk over the stale one. It does this without adding any extra computation during runtime or waiting for new observations.

Dev: That’s a big part of it, isn't it? The core claim is that this changes the policy's action preference under stale inputs without needing any runtime repair mechanisms, which is pretty neat for deployment safety.

The paper's improvements: Rosa: One of the main things they point out as an improvement is that DEFLECT extends the usable delay envelope of asynchronous VLA control, specifically improving high-latency success by plus six point four percentage points in simulation and a plus four point six-point gain at the longest delay on a real-scale VLA <ref:2605.19294#pg2>.

Dev: That six point four increase over VLASH in simulation is substantial, especially since VLASH’s performance drops significantly as the delay gets longer, which is where we usually struggle most with asynchronous systems.

Taro: What I find interesting about their method is that it's a dynamics-aware, variance-preserving correction. They suggest that the correction applies larger adjustments to tasks with sharp transient dynamics and smaller corrections for things that are slower or more periodic.

Rosa: That means the method isn't just applying a uniform fix across all situations; it’s actually tuning how much it corrects based on the kind of motion it’s dealing with, which helps avoid mode collapse.

Dev: And they keep the expert action manifold intact using an SFT anchor term, which is important because we don't want this preference optimization to drift too far from what we already know works for real-world safety.

Conclusion: Rosa: To wrap up, DEFLECT shows that by using temporal counterfactual supervision, we can teach these asynchronous VLA policies to prefer execution-time actions over stale observations. It achieves this by constructing preference pairs from offline expert data and scoring them against the deployment input it will actually see.

Dev: The implication for us in real-time control is that we don't need to add complex runtime repair or waiting mechanisms to handle latency; we can just train the policy to be inherently better at handling stale inputs when deploying it.

Taro: For me, the fact that the net DPO contribution is concentrated at higher delays shows that this preference alignment concentrates its benefit precisely in those delay regimes where fresh and stale observations produce different actions.

Rosa: So, DEFLECT isn't just a tuning trick; it seems to be a general counterfactual-supervision recipe where both ideal and deployable conditions can be scored under the deployment context to provide a zero-cost preference signal.

Dev: It’s still important to remember their limitation: they flag that the fresh/stale preference signal is weakest at low delays because those two observations are nearly identical, so you get most of the DPO lift in the high-delay regime.

Taro: And they also noted that PFM underperforms VLASH when the delay is two or more, confirming that generic expert-vs-policy alignment can actually make things worse when applied to stale observations.

Rosa: So, DEFLECT improves high-delay execution by six point four percentage points over VLASH on Kinetix and a four point six point gain on LIBERO. It’s a concrete improvement over current asynchronous baselines in challenging scenarios.

Dev: We're looking forward to seeing how this kind of preference alignment applies to other real-time loops, but for now, DEFLECT gives us a solid way to extend the usable delay envelope for these VLA systems.