DynamicVLA: A Vision-Language-Action Model for Dynamic Object Manipulation

Manipulating dynamic objects remains an open challenge for Vision-Language-Action (VLA) models, which struggle in dynamic scenarios requiring rapid perception, temporal anticipation, and continuous control. DynamicVLA is presented as a framework that integrates temporal reasoning and closed-loop adaptation through three key designs to address the perception–execution gap inherent in existing VLA models.

The gist: DynamicVLA is a framework for dynamic object manipulation that integrates temporal reasoning and closed-loop adaptation through three key designs: 1) a compact 0.4B VLA using a convolutional vision encoder for spatially efficient, structurally faithful encoding, enabling fast multimodal inference; 2) Continuous Inference, enabling overlapping reasoning and execution for lower latency and timely adaptation to object motion; and 3) Latent-aware Action Streaming, which bridges the perception–execution gap by enforcing temporally aligned action execution.

The DynamicVLA Architecture

The framework is built around a compact 0.4B VLA model designed for fast and spatially efficient multimodal reasoning. It utilizes a convolutional vision encoder, FastViT, for efficient spatial compression and stronger structural preservation, which avoids quadratic token growth when processing multiframe visual inputs. The language backbone is truncated to its first 16 layers of SmolLM2-360M to significantly reduce inference latency with minimal impact on multimodal reasoning. Action generation is handled by a Diffusion-Based Action Expert, instantiated as a conditional Flow Matching Transformer, trained using the objective where it learns to match the denoising vector field.

Continuous Inference

This design addresses inter-chunk waiting by enabling overlapping reasoning and execution for lower latency and timely adaptation to object motion. Unlike prior models where inference is triggered only after a sequence is fully executed, Continuous Inference overlaps prediction and action execution, allowing actions from the current sequence to be executed while the next one is being inferred. This eliminates inter-chunk waiting under dynamic object motion.

Latent-aware Action Streaming

This mechanism restores temporal alignment by enforcing temporally consistent control despite inference delay, which manifests as a Perception–Execute Gap. It resolves this through an explicit execution strategy: actions in the current sequence corresponding to timesteps earlier than the observation time are discarded as outdated, and execution proceeds with the subsequence of newer actions. Furthermore, where action chunks overlap, actions from the newer sequence are prioritized, overwriting those from At, ensuring adaptation to the most recent environment state.

The Dynamic Object Manipulation (DOM) Benchmark

To fill the missing foundation of dynamic manipulation data, a large-scale benchmark was introduced. DOM features 200K synthetic episodes across 2.8K scenes and 206 objects, and enables fast collection of 2K real-world episodes without teleoperation. The benchmark organizes evaluation along three principal dimensions: Interaction (Closed-loop reactivity, Dynamic adaptation, Long-horizon sequencing), Perception (Visual understanding, Spatial reasoning, Motion perception), and Generalization (Visual generalization to unseen objects/scenes and Motion generalization).

Experimental Evaluation

DynamicVLA was evaluated across dynamic manipulation tasks using the DOM benchmark. Extensive evaluations demonstrated remarkable improvements in response speed, perception, and generalization, outperforming baselines such as SmolVLA by significant margins across all interaction settings. Ablation studies confirmed that the 360M model strikes the optimal balance between inference efficiency and model capacity, yielding the highest overall performance in dynamic object manipulation. The dominant failure mode identified is temporal misalignment between observation and action execution.

Ablation Findings

Ablation studies revealed that increasing LLM depth leads to a modest increase in latency, which can be amortized by Continuous Inference and Latent-aware Action Streaming. Aggressively truncating the backbone improves inference speed but degrades success rate due to reduced reasoning capacity. The integration of CI and LAAS into existing models, such as SmolVLA, shows consistent performance improvements under moderate inference latency, whereas larger backbones in models like π0.5 limit the effectiveness of these mechanisms due to high underlying latency.

Future Work Directions

Future work should focus on extending dynamic manipulation to multi-stage tasks with persistent object motion, integrating planning and memory while maintaining real-time constraints. Additionally, extending the model to handle non-rigid or fluid dynamics remains an open challenge for both the VLA models and data pipelines.

Acknowledgments

The work was supported by the Singapore Ministry of Education under its Academic Research Fund Tier 2 (MOE-T2EP20221-0012, MOE-T2EP2023-0001), and by cash and in-kind contributions from NTU S-Lab and industry partner(s).

References

[1] DeepSeek AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv, 2501.12948, 2025.

Based on the DynamicVLA paper, here are specific improvements that can be made to AI systems, along with what those improved systems will be able to do:

1. Improve Real-Time Responsiveness and Stability in Dynamic Environments by Integrating Latency-Aware Action Streaming (LAAS).

2. Improve Perception-Execution Alignment by Implementing Continuous Inference (CI).

3. Enhance Generalization to Unseen Objects, Scenes, and Motion Regimes by Utilizing the Dynamic Object Manipulation (DOM) Benchmark for Training.

4. Increase Model Efficiency and Throughput while Maintaining High Performance by Employing a Compact 0.4B VLA Architecture with FastViT Vision Encoder.

5. Enable Robust Closed-Loop Reactivity to Fast-Moving Objects and Abrupt Changes in Motion through Temporal Reasoning Integration (e.g., using the specific temporal context window of two frames: 3 and 2).

6. Achieve Superior Spatial Reasoning and Motion Perception by Training Models on Multi-faceted DOM Sub-tasks (Interaction, Perception, Generalization).

7. Develop Autonomous Data Collection Pipelines for Dynamic Manipulation by Utilizing the Automated Simulation and Real-world Data Collection Framework (using Isaac Sim and a real-world simulator pipeline) to generate 200K synthetic and 2K real-world episodes without reliance on slow teleoperation.

8. Create More Efficient, Adaptable VLA Models by Performing Ablation Studies that identify the optimal balance between LLM Backbone Capacity (e.g., 16 layers vs. 32 layers) and Inference Latency, leading to smaller, faster models with high success rates.

This improved AI system can perform the following specific actions:

1. Navigate and interact with objects in real-time environments where objects are moving at varying speeds (e.g., grasping a rolling coffee can or placing an object onto a frisbee) while maintaining stable, closed-loop control that adjusts immediately to sudden disturbances.

2. Execute complex, long-horizon manipulation sequences (e.g., gathering multiple items into a container) without stalling due to inference delays between action steps.

3. Identify and grasp objects in cluttered or visually similar scenes (e.g., picking up a specific tennis ball among several others) by accurately perceiving and grounding visual cues in dynamic, changing environments.

4. Generalize its manipulation skills to novel objects with unseen appearances, novel 3D scenes, and unpredictable motion patterns (e.g., grasping an apple with an irregular trajectory), demonstrating robustness beyond the training distribution.

5. Operate reliably in complex physical settings (like real-world Franka or PiPER arms) where human reaction times are too slow to track fast-moving objects, effectively acting as a high-speed, closed-loop control system.

6. Perform precise spatial reasoning tasks, such as placing an object relative to a specific location or within a dynamically defined boundary (e.g., placing a bottle onto a blue tape) despite continuous object motion and visual noise.

7. Self-sustain learning by autonomously generating large, diverse datasets for dynamic manipulation through automated simulation and real-world sensing, reducing the dependency on costly human teleoperation for data labeling.