Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models

arXiv:2608.27550 · cs.RO, cs.CV · Submitted 2026-08-27 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa, and with me are Dev and Taro, guest researcher.

Dev: Today's paper: "Beyond Data Scaling".

Rosa: As a fastidious and diligent AI researcher, I have meticulously analyzed both provided texts from arXiv.

Dev: First, who's behind it and why it matters.

Paper summary: Rosa: So we're talking about this paper called "Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models," and the main idea is that just feeding more robot data isn't the whole answer for making these VLA models better.

Dev: Exactly, Rosa; the authors argue that because robot trajectories are inherently harder to scale than web data due to needing physical collection, we have to focus on converting those limited trajectories into reusable knowledge instead of just fitting actions.

Taro: I'm interested in what they claim about this representation-centric approach and how it tackles the challenges inherent in training these models.

Rosa: The core thesis of VLAct is that continued pre-training should be focused on distilling generalizable visual and action knowledge from robot trajectories, which offers a path forward when you're constrained by the physical collection process.

Dev: It claims they propose VLAct, a VLA-oriented VLM backbone trained with a specific recipe that starts from an existing pre-trained VLM and trains on broad, heterogeneous, multi-embodiment robot data before any task-specific fine-tuning happens.

Taro: So instead of just scaling up the dataset volume, they're focusing on how the model learns to represent actions across different situations and robots in a meaningful way.

Rosa: That's right; VLAct aims to do three things: preserve the broad VLM prior, prevent over-specialization to a single action head, and encourage shared action semantics between different embodiments.

Dev: To address that first point, they use shallow-layer protection and caption mixing to stop the model from eroding its general vision-language features learned initially when training on larger datasets.

Taro: That makes sense; if we just train it on robot data, it could forget how to interpret complex visual scenes or language instructions that came from web corpora.

Rosa: They also tackle the issue of single action head specialization by using multi-head continuous action co-supervision with distinct heads like OFT, PI, and GR00T during the training process.

Dev: And for the third problem, which is about different robot platforms not sharing action concepts well, they employ a partially unified cross-embodiment action layout where supervision only happens when actions are physically comparable.

Taro: That’s interesting; it suggests that by focusing on physically relevant similarities in action spaces, we can force the model to learn more transferable semantics across different physical systems.

Rosa: The technical design specifics also include a fixed twenty-dimensional vector for continuous action heads where dimensions change based on the embodiment's role, and they use wrap-aware loss to handle continuous joint movements correctly.

Dev: That wrap-aware loss sounds critical from an engineering standpoint because you don't want artifacts when dealing with absolute joint angles in a real robot system; that handles the continuity issue directly.

Taro: It sounds like they are building a framework that tries to maintain robustness across different action spaces while still learning from the physical constraints of the embodied data.

Paper summary: Rosa: The empirical results show that VLAct consistently outperforms generic VLM backbones and naive VLA-pretrained baselines across various challenging benchmarks, achieving success rates like eighty-two point six percent on LIBERO-Plus <ref:2608.27550#pg0>.

Dev: And perhaps more importantly for us engineers, they demonstrate superior data efficiency by beating the full-data GR00T-N1 point 6 baseline using only twenty percent of downstream trajectories from the RoboCasa-GR1 dataset.

Taro: That data efficiency finding is significant; it shows that this representation-centric method can extract a lot of useful knowledge without needing the massive amounts of raw robot data that naive scaling methods require.

Rosa: The paper's main conclusion is that VLAct learns reusable action representations rather than just overfitting to specific robot embodiments observed during pre-training, which strongly supports their central claim about generalization.

Dev: It implies that even with limited physical data, we can build models capable of transferring those learned skills to new, unseen environments and different robot platforms effectively.

Taro: The implication is that the path toward generalist robotics isn't just about bigger datasets; it's about designing better training recipes that prioritize knowledge distillation and representation quality over sheer data volume.

Rosa: So, the paper suggests a shift in focus for continued pre-training from simply scaling up to a more targeted, representation-centric approach for VLA models.

Dev: It gives us a concrete set of architectural solutions—like multi-head supervision and layout unification—that we can actually implement into our control loops without needing to collect petabytes of new robot data immediately.

Taro: Thinking about the future, if this holds up in real-world experiments, it means we might see VLA models deployed in environments that are slightly different from the training setups they observed.

Rosa: That's what we need to watch for; whether this works long-term outside the lab and how robust these learned representations are under novel conditions is a big question for field robotics.

Dev: From my side, I'm focused on the loop rate and latency implications of these complex representation layers, so understanding how efficient VLAct inference would be in a real-time deployment is something we need to investigate further.

Taro: And for autonomy research, this method opens up possibilities for systems that need to adapt quickly to novel physical interactions without needing a complete retraining cycle every time they encounter a new type of robot or task.

Rosa: We'll keep tracking the progress on this paper and see how these findings translate into tangible improvements for autonomous systems operating in the physical world.

Dev: Indeed, focusing on those architectural details and real-world performance metrics will be key to seeing if VLAct delivers what it promises beyond simulation success rates like the ninety-two point five percent seen on RoboTwin two point zero.

Taro: It really feels like a step toward making VLA models truly versatile agents that don't get stuck in narrow operational domains because of how they were initially trained and embodied.

Rosa: We'll continue to follow this work closely as it moves from the paper to actual system implementation in the field.

Conclusion: Rosa: So, we're wrapping up our discussion on "Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models," and I just want to touch on what that title actually suggests about the work by those authors.

Dev: It really does; that title immediately signals a shift away from simply throwing more data at the problem and toward a more intelligent way of learning.

Taro: I think it points toward a focus on *how* the model learns to represent actions, which is crucial when you're dealing with physical systems.

Rosa: Exactly, and the authors are pushing that idea that we need better recipes for continued pre-training rather than just bigger datasets to see real progress.

Dev: That means they’ve found a way to distill useful knowledge from robot trajectories in a more efficient manner than previous methods allowed.

Taro: And if this method is successful, it could mean we can deploy more capable agents into complex physical environments with less initial data collection overhead.

Rosa: It opens up the possibility that VLA models won't just be good at tasks they saw during training but will actually generalize to new physical setups.

Dev: I'm curious about how long this learned representation stays robust when the robot encounters something completely different outside of the lab setting.

Taro: That’s a tough question, Dev; we need to see if these learned action semantics hold up when the dynamics or object shapes change significantly.

Rosa: And that brings us to a big question for field robotics: can these models operate effectively in real-world scenarios without constant retraining cycles?

Dev: We'll have to look closely at the computational overhead of those representation layers; if inference is too slow, even the best learned knowledge won't be useful in real-time control.

cs.RO, cs.CV

Submitted: 2026-08-27

Updated: 2026-10-06

Comments: All models and training pipelines are publicly available at https://starvla.github.io/VLAct

Project page: https://starvla.github.io/VLAct

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: As a fastidious and diligent AI researcher, I have meticulously analyzed both provided texts from arXiv.

Key concepts

Shallow-Layer Protection
This technique preserves general vision and language knowledge by freezing the lower layers of the vision encoder and language model during continued pre-training. This prevents the model from forgetting broad priors learned from large web corpora while it learns specific action details from robot data.
Multi-Head Continuous Action Co-supervision
Instead of training on just one action output, VLAct uses multiple distinct heads (like OFT, PI, and GR00T) simultaneously. This forces the model to learn a more versatile representation that is not overly specialized for a single robot's movement style.
Unified Action Representation
The model emits a fixed-size action vector at every timestep, but its dimensions change based on the robot's role. This ensures that low-level action conventions are consistent across different robotic platforms, making the learned knowledge more transferable.

Terminology

Summary

As a fastidious and diligent AI researcher, I have meticulously analyzed both provided texts from arXiv. My primary objective is to synthesize these disparate pieces of information into a comprehensive, detailed summary of the paper titled Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models.

The first text provides the core technical contribution and empirical results of the proposed method, while the second text offers context regarding limitations, broader impact, and illustrative examples from related work.

Here is my detailed synthesis of the paper's content:

The paper introduces VLAct, a novel approach to continued pre-training for Vision-Language Action (VLA) models, positioning it as a representation-centric method designed to distill reusable visual and action knowledge from robot trajectories. The central thesis is that effective continued pre-training can significantly improve downstream performance under fixed resource budgets, offering an independent axis of progress beyond mere data scaling.

VLAct frames continued pre-training not as a monolithic end-to-end update, but as a process of distilling robot trajectories into generalized knowledge within the backbone. The authors identify three critical failure modes in naive VLA continued pre-training and propose specific architectural solutions to counteract them:

  1. Erosion of Broad VLM Priors: Naive methods risk eroding the general vision-language features learned during initial VLMs due to the narrow nature of robot trajectories compared to web-scale corpora.
  • VLAct Solution: It preserves this prior through shallow-layer protection, which involves freezing the vision encoder and lower layers of the Language Model (LLM). Furthermore, it employs caption mixing, where robot trajectories are mixed with caption data during pre-training.
  1. Over-Specialization to a Single Action Head: Training solely on one action head can cause the backbone to over-specialize its decoding geometry for that specific output space, hindering transferability to other heads.
  • VLAct Solution: This is mitigated via multi-head continuous action co-supervision, utilizing distinct heads such as OFT, PI, and GR00T.
  1. Weak Cross-Embodiment Sharing: Embodiment-specific output spaces can prevent the learning of shared action semantics across different robot platforms.
  • VLAct Solution: The model encourages this sharing through a partially unified cross-embodiment action layout, where supervision is only applied in areas where actions are physically comparable (e.g., matching gripper dimensions).

The paper details specific, carefully designed components to ensure robustness and generalization:

  • Unified Action Representation: For continuous action heads, a fixed 20-dimensional vector is emitted at each timestep. The dimensions of this vector are dynamically assigned based on the embodiment's role (e.g., absolute joint angles for bimanual arms versus delta end-effector poses for single-arm arms). This design ensures that low-level action conventions are preserved across different robotic platforms.

  • Wrap-aware Loss: To handle continuous joint movements accurately, a wrap-aware loss is implemented on absolute joint angles. This loss wraps the angles into the canonical range [-pi, pi] and applies a residual wrapping loss to correct any artifacts near this boundary.

The experiments demonstrate that VLAct consistently outperforms generic VLM backbones and naive VLA-pretrained baselines across various challenging benchmarks:

  • Performance Metrics: VLAct achieves high success rates on multi-embodiment simulation benchmarks, real-world robot experiments, and unseen-embodiment transfer.

  • On LIBERO-Plus, VLAct surpasses large industrial systems like ABot-M0, achieving an 82.6% success rate.

  • On RoboTwin 2.0, it reaches a 92.5% success rate.

  • On the recently published RoboDojo simulation benchmark, VLAct achieves an average score of 10.66 and a 7.60% success rate, ranking sixth among all policies by success rate and outperforming designated World-Action-Model (WAM) entries on both metrics.

  • Data Efficiency: A crucial finding is the model's efficiency: VLAct outperforms the full-data GR00T-N1.6 baseline (47.6% success) using only 20% of downstream trajectories from the RoboCasa-GR1 dataset, demonstrating superior data efficiency.

  • Generalization: The results strongly support the central claim that VLAct learns reusable action representations rather than overfitting to specific robot embodiments observed during pre-training.

Improvements for AI systems

As a fastidious researcher, I have meticulously analyzed the StarVLA: Beyond Data Scaling paper. The core contribution of VLAct is a representation-centric continued pre-training recipe that addresses the fundamental bottleneck in scaling Vision-Language-Action (VLA) models: converting sparse, embodied trajectories into transferable visual-action knowledge.

Here are the specific improvements and capabilities this system enables for AI agents:

Core Architectural & Training Improvements (VLAct Recipe):

  1. To build a robust VLA backbone that generalizes across tasks, embodiments, and action heads without overfitting to narrow trajectory distributions.

  2. Preserve the broad vision-language prior inherited from massive web-scale VLMs by utilizing a two-pronged strategy:

  3. Freeze the entire Vision Encoder and the lower half of the LLM layers during continued pre-training (Shallow-Layer Protection). This preserves general perceptual and early vision-language alignment features.

  4. Mix robot trajectories with diverse auxiliary non-action data to provide representation anchoring and feature diversification. The primary anchor is image captioning, which provides dense semantic supervision over objects, attributes, relations, and scene context.

  5. Employ multi-head continuous action co-supervision: Train the shared backbone with multiple continuous heads (OFT for regression, PI for flow-matching, GR00T for dual-system flow-matching). This forces the representation to be head-agnostic and prevents decoder lock-in.

  6. Implement a partially unified cross-embodiment action space: Use a single shared action head with an action space that is unified only along physically comparable dimensions (e.g., sharing gripper commands across Franka and AgileX, while keeping arm kinematics separate). This ensures shared supervision where semantics align without forcing incompatible degrees of freedom into the same coordinate system.

  7. Apply advanced action representation regularization: Use a wrap-aware loss for periodic joint angles to construct a unified joint space and mitigate ambiguity near angular boundaries, ensuring consistent numerical representations across different datasets.

  8. Implement a unified data cleaning pipeline: Remove invalid task names and filter action chunks based on an invalid-step ratio to ensure reliable supervision signals are used, regardless of the heterogeneity of real-world robotic data sources.

Enhanced Capabilities of the Improved AI System (VLAct):

  1. Superior Generalization Across Embodiments (Cross-Embodiment Transfer):

  2. The system can effectively transfer learned skills from a humanoid embodiment never seen during pre-training (e.g., GR-1) to a completely new, unseen robotic platform (e.g., ARX X5), achieving high performance even with minimal downstream data (only 20% of trajectories required).

  3. Robust Performance Under Distribution Shifts:

  4. The policy remains highly stable and performs well under significant visual and scene changes (Out-of-Domain generalization) when encountering novel objects or extended, unseen task sequences, demonstrating superior visual-semantic grounding compared to non-pretrained baselines.

  5. Enhanced Long-Horizon Planning and Task Logic:

  6. The system excels at multi-stage manipulation tasks (e.g., table cleaning, complex assembly), maintaining task logic and sequence integrity even when facing object substitutions or extended sequences, unlike baselines that often fail to complete necessary intermediate steps (e.g., skipping the scooping phase).

  7. Advanced Dual-Arm Coordination:

  8. The system demonstrates superior coordination in bimanual tasks (e.g., folding pants, towel folding) by producing more synchronized motion and maintaining stable contact with deformable objects, overcoming velocity mismatches that plague simpler models.

  9. Optimized for Heterogeneous Data Integration:

  10. The model can effectively absorb and learn from a mixture of data collected under different collection paradigms (e.g., handheld UMI interfaces) and robot platforms (Franka vs. AgileX), resulting in a single, robust action representation rather than learning dataset-specific artifacts.

  11. Versatility Across Action Heads:

  12. The system provides a reusable backbone that can be fine-tuned with any desired downstream action head (FAST, OFT, PI, or GR00T) without suffering from severe decoder lock-in, allowing users to select the optimal action modality for their specific control needs.

Abstract

Scaling robot data is crucial for building generalist Vision-Language-Action (VLA) models, yet robot trajectories are harder to scale than web-scale image-text data because embodied collection is costly and sparsely covers the physical world. This makes representation quality a central bottleneck: under a fixed robot-data budget, continued pre-training must turn limited trajectories into transferable visual-action knowledge rather than merely fit actions. We propose VLAct, a VLA-oriented VLM backbone trained on broad, heterogeneous, multi-embodiment robot data before task-specific fine-tuning. VLAct preserves the broad VLM prior and encourages shared action semantics across embodiments through VLM-prior preservation, multi-head continuous action co-supervision, and a partially unified cross-embodiment action layout, while allowing task-specific action heads during fine-tuning. Across simulation, real-world, and unseen-embodiment transfer, VLAct consistently improves downstream performance under fixed fine-tuning protocols. On LIBERO-Plus and RoboTwin 2.0, VLAct surpasses industrial VLA systems including ABot-M0 and LingBot-VLA, achieving success rates of 82.6% and 92.5%. On RoboDojo, VLAct ranks sixth among all policies by success rate and outperforms all explicitly designated world-action model (WAM) entries on both metrics. Most notably, on RoboCasa-GR1, an unseen humanoid embodiment, VLAct using only 20% of downstream trajectories outperforms the full-data GR00T-N1.6 baseline. These results are obtained using fully open-source data and only a 16-GPU training setup, showing that representation-centric continued pre-training can deliver highly competitive performance under a modest compute budget and is an important independent axis of VLA progress beyond data scaling.

Sources

Related papers