Lost in Reconstruction: Aligning Action Representations with Language in Vision-Language-Action Models

The paper introduces SALT, a Semantically ALigned action Tokenizer, to address the problem that action representations in vision-language-action models (VLAs) are typically optimized for reconstruction under L1/L2 losses in raw action space, where numerical proximity need not reflect linguistically meaningful distinctions. The authors show on BridgeV2 that action trajectories contain verb-grounding information beyond visual state changes, and that reconstruction-only discrete tokenization systematically erodes this information. To address this, SALT augments a VQ-VAE-style tokenizer with an auxiliary objective requiring a frozen vision-language model to recover the episode instruction from quantized action latents. Policies trained with SALT achieve 71.9% average success in SimplerEnv, compared with 42.7% for a reconstruction-only VQ-VAE tokenizer and 31.2% for FAST. SALT also develops verb-specialized codes while maintaining reconstruction fidelity. These results show that robot action trajectories provide a source of language grounding and that preserving this structure in action representations can substantially improve language-conditioned control.

The paper establishes three findings: (i) Action trajectories contain verb-grounding information that visual outcomes alone do not capture. The authors decompose verb information into action goals (the visual change between episode endpoints) and motion dynamics (the 7-DoF action trajectory). Each carries verb information not captured by the other as verbs are grounded not only in resulting state changes, but also in motion profiles, such as contact patterns and gripper timing. (ii) Reconstruction-only action tokenization systematically erodes verb-grounding signals. Across Bin, VQ-VAE, and FAST tokenizers, mutual information between verbs and tokenized action representations decreases, with larger losses under stronger compression. Downstream language-conditioned policy training does not fully recover the missing structure. (iii) Semantically aligned tokenization improves task performance and produces more linguistically meaningful action representations.

In Diagnostic 1, the authors estimate I(Y; X) = H(Y) − H(Y X) from the held-out cross-entropy of matched Transformer classifiers under 5-fold stratified cross-validation. The motion condition receives the 7-DoF action sequence, the goal condition receives frozen DINOv2-S representations of the first and last frames, and the combined condition receives both. Table 1 shows that action goals and motion dynamics provide complementary information about verbs. In aggregate, the combined representation contains more verb information than either modality alone: motion contributes an estimated 0.059 bits beyond the visual goal representation, while action goals contribute 0.282 bits beyond motion. Visual state change explains more unique verb information overall, but does not exhaust the information available in the trajectory. The motion-specific contribution is concentrated among verbs whose endpoint states can appear similar despite systematic differences in how the action is executed. For example, move and put together account for approximately two-thirds of the estimated motion-unique signal, while the less frequent verb push also contributes positively.

In Diagnostic 2, the authors apply the verb-decoding analysis to token sequences produced by each tokenizer across compression levels. Figure 3 reports mutual information between verbs and tokenized actions as a function of bitrate, with the continuous trajectory as a reference. Across all three families, tokenized representations retain less verb information than the continuous actions, and the gap generally widens as compression increases. These results show that the reconstruction-only objective does not guarantee preservation of language grounding signals in compressed action representations. A small error in Euclidean space could be consequential for verbs, and larger variations could be incidental and semantically equivalent, bottlenecking downstream VLA performance.

SALT intervenes exclusively in the tokenizer-training stage: it changes how the action vocabulary is learned, while VLA training and the policy’s use of the tokens remain unchanged. SALT augments the standard VQ-VAE objective with a language-alignment loss, L = Lrecon + LVQ + λLalign, where Lrecon is the action reconstruction loss, LVQ comprises the codebook and commitment losses, and Lalign encourages the tokenizer to preserve information predictive of language. For alignment, an episode is divided into M action chunks, yielding quantized latents q1:M. The tokenizer latent dimension is matched to the language-model embedding dimension, and each latent is converted into a soft prefix embedding, pi = g qi + PE(i), where g is a learned scalar gain and PE(i) is a positional encoding. The resulting prefix sequence is provided, together with a short textual prompt, to a frozen pretrained language model, which predicts the episode instruction. The alignment objective is Lalign = −(1/L) Σ log pLM (wt w<t, P, s). The language model remains frozen, while gradients propagate through the input embeddings and the straight-through quantizer into the tokenizer encoder and codebooks. Because it operates directly on free-form instructions, it requires neither a predefined verb inventory nor a separate text encoder or contrastive negative pairs. After tokenizer training, the language model is discarded and the tokenizer is frozen.

In experiments, SALT is evaluated as a drop-in action tokenizer for miniVLA, a Prismatic-style VLA with a Qwen2.5-0.5B backbone. All policies are trained on BridgeV2 from the base vision-language checkpoint for 15k gradient steps at a global batch size of 128. The action tokenizer is the only component that differs across conditions. SALT and VQ-VAE share the same residual-VQ architecture (8-timestep chunks; 7 residual groups of 256 codes each) and the same training data, and differ only in the language-alignment loss. FAST is fitted on the same BridgeV2 chunks with vocabulary 1,024 and ≈7 tokens per chunk. Rollout success is measured in SimplerEnv’s visual-matching WidowX suite: four tabletop tasks, 24 episodes each (96 rollouts per policy), with open-loop execution of 8-step action chunks.

Table 2 reports rollout success. SALT reaches 71.9%, against 42.7% for VQ-VAE and 31.2% for FAST, and leads on every individual task, with the largest margins on the two most difficult (stack: 70.8 vs. 33.3; eggplant: 79.2 vs. 33.3). The SALT-vs-VQ-VAE comparison isolates the alignment loss: the two tokenizers share architecture, capacity, data, and the entire VLA training recipe, so the 29.2-point gap is attributable to how the action vocabulary is partitioned, not to how much it compresses.

Language alignment transfers beyond the tokenizer. Although the alignment objective supervises only the tokenizer’s latent representations, its effects propagate to both the discrete token IDs produced by the tokenizer and the action-token embeddings learned by the downstream VLA (Table 3). SALT achieves the highest verb decodability at both stages, reaching 39.1 macro-F1 on token IDs (vs. 37.3 for VQ-VAE and 30.3 for FAST) and 43.7 on learned action embeddings (vs. 38.3 and 36.3). Moreover, its token-ID accuracy (58.7%) matches the continuous-action reference (58.0%). Semantic alignment preserves reconstruction fidelity. At the matched compression rate of seven tokens per eight-step chunk, SALT’s held-out reconstruction error remains close to that of VQ-VAE (Table 3). The rank correlation of one-vs-rest effect sizes across 63 interpretable trajectory features remains at least 0.92 (vs. 0.96 for VQ-VAE).

Verb-specialized codes emerge under semantic alignment. Figure 4 shows that SALT develops highly selective codes associated with particular verbs, whereas VQ-VAE distributes the same trajectories across semantically mixed codes and FAST largely reflects the corpus verb distribution. Figure 5 illustrates this effect for representative flip and turn codes. Interestingly, the turn code also captures instructions phrased as “lever vertical to front,” grouping semantically equivalent actions despite different wording. A probe-free majority-vote lookup based solely on code–verb co-occurrence achieves higher held-out accuracy for SALT than VQ-VAE for every individual residual group and every cumulative group prefix; for example, using only the first two residual groups yields 46.3% episode-level accuracy for SALT versus 43.6% for VQ-VAE (McNemar p =.011), with FAST reaching 35.0%.

The paper concludes with limitations: current language-conditioned robotics datasets exhibit limited linguistic diversity (only 17 verb classes after filtering); SALT currently applies only to tokenizers with a learnable latent representation; experiments use a relatively small VLA (0.5B parameters), a single training dataset, and simulation-based evaluation; and the paper does not establish a causal mechanism relating semantically organized action codes to improved downstream policy performance.

Improvements to AI Systems:

1. Add a language-alignment loss to action tokenizers in VLA models.

• Modify the tokenizer training objective to include a frozen language model that predicts the episode instruction from quantized action latents (as in SALT).

• This preserves verb-grounding information (e.g., flip, push, turn) that reconstruction-only losses (L1/L2, VQ-VAE, FAST) systematically erode.

• The improved system can execute language-conditioned manipulation tasks with higher success (e.g., 71.9% vs. 42.7% on SimplerEnv) by retaining motion dynamics (contact patterns, gripper timing) that visual state changes alone do not capture.

2. Use multi-modal verb-information decomposition to enrich action representations.

• During tokenizer training, explicitly separate verb information into action goals (visual endpoint changes) and motion dynamics (7-DoF trajectories).

• Train the tokenizer to preserve both, not just the visual outcome, so that verbs like move vs. put (which can have similar endpoints but different motion profiles) are distinguished.

• The improved system can better generalize to novel instructions with subtle motion differences, reducing failures in tasks where visual outcomes are ambiguous.

3. Replace reconstruction-only compression with semantically aligned compression.

• In any VLA tokenizer, add a loss term that maximizes mutual information between quantized latents and free-form language instructions (no predefined verb list or contrastive pairs needed).

• This ensures that small Euclidean errors in action space do not erase linguistically meaningful distinctions, while larger variations that are semantically equivalent are tolerated.

• The improved system maintains reconstruction fidelity (error close to VQ-VAE) while achieving higher verb decodability (43.7 macro-F1 on learned action embeddings vs. 38.3 for VQ-VAE), leading to more robust policy learning.

4. Leverage emergent verb-specialized codes for interpretable and transferable action vocabularies.

• After training with SALT, use the learned codebook to create a majority-vote verb lookup (based on code–verb co-occurrence) for zero-shot action classification or debugging.

• The improved system can group semantically equivalent actions (e.g., lever vertical to front and turn) under the same code, enabling better cross-task transfer and reducing the need for large, diverse datasets.

5. Apply the alignment loss at multiple stages of the VLA pipeline.

• Beyond tokenizer training, fine-tune the downstream action-token embeddings with a similar language-alignment objective (e.g., predict instruction from embedding averages).

• This propagates semantic structure into the policy’s internal representations, improving language-conditioned control even when the tokenizer is frozen.

• The improved system can achieve higher success on difficult tasks (e.g., stacking, eggplant manipulation) by making the policy more sensitive to verb-specific motion requirements.

6. Use the diagnostic mutual-information metric as a training-time regularizer.

• During tokenizer training, periodically estimate I(verb; tokenized actions) via a lightweight classifier and add a penalty if it drops below a threshold.

• This prevents over-compression that erodes language grounding, even when the dataset has limited verb diversity (e.g., 17 classes).

• The improved system can maintain language-aligned representations across different compression levels, avoiding the performance cliff seen with reconstruction-only tokenizers at higher bitrate reductions.

Action verbs describe not only the physical outcomes of actions, but also how those actions are performed. Yet action representations in vision-language-action models (VLAs) are typically optimized for reconstruction under L1/L2 losses in raw action space, where numerical proximity need not reflect linguistically meaningful distinctions. On BridgeV2, we show that action trajectories contain verb-grounding information beyond visual state changes, and that reconstruction-only discrete tokenization systematically erodes this information. To address this problem, we introduce SALT, a Semantically ALigned action Tokenizer that augments a VQ-VAE-style tokenizer with an auxiliary objective requiring a frozen vision-language model to recover the episode instruction from quantized action latents. Policies trained with SALT achieve 71.9% average success in SimplerEnv, compared with 42.7% for a reconstruction-only VQ-VAE tokenizer and 31.2% for FAST. SALT also develops verb-specialized codes while maintaining reconstruction fidelity. These results show that robot action trajectories provide a source of language grounding and that preserving this structure in action representations can substantially improve language-conditioned control.