BooST: Bridging Semantics and Motions for Efficient Skill Transfer

BooST: Bridging Semantics and Motions for Efficient Skill Transfer introduces a two-stage framework for learning a unified skill representation that captures both high-level semantic intent (what) and low-level motion dynamics (how) to enable efficient skill transfer to real robots. The paper states that "existing methods typically satisfy only a subset of these properties, as they capture either high-level semantic intent (what) or low-level motion dynamics (how). This incomplete skill transfer yields weak priors for policy learning, thereby demanding substantial in-domain data for downstream adaptation."

The framework consists of two stages. In Stage I (Unified Skill Pretraining), BooST leverages a cross-modal VQ-VAE to capture both semantic intent and motion dynamics, yielding a unified skill representation. This is achieved through two complementary pathways: a visuo-linguistic pathway that extracts semantic intent from visual observations and language instructions using a pretrained CLIP model and a cross-attention mechanism, and an action pathway that encodes motion dynamics from action trajectories, grounding the learned semantics in executable behaviors. The two pathways are optimized alternately, and the model uses residual vector quantization with a codebook of size K=16 and 2 quantization levels. The pretraining objective is a weighted sum of reconstruction losses from both pathways plus a regularization term: L pretrain = λ1∥at:t+H − âvl∥2 + λ2∥at:t+H − âact∥2 + λ3∥fenc,vl − fenc,act∥2, with λ1=3, λ2=1, λ3=0.5.

In Stage II (Downstream Adaptation), BooST distills this representation into a lightweight policy for efficient downstream adaptation to new tasks. The downstream objective is L downstream = E[−log pψ(ztq Ot−L:t, l)] + α E[−log πθ(at:t+H sg(zt), Ot−L:t)], where the first term is the prior distillation loss and the second is the policy behavior cloning loss. The skill prior pψ and low-level policy πθ are implemented as small Transformer models, with the policy head following BAKU.

The paper reports extensive experiments in simulation (LIBERO benchmark) and real-world settings (UR3 robot with Robotiq 2F-85 gripper), with pretraining on the large-scale DROID dataset (76k teleoperated trajectories with a Franka Emika Panda arm). Key results include:

1. Few-shot adaptation (Q1): On LIBERO benchmarks with 50, 20, and 10 demonstrations, BooST consistently outperforms all baselines (Diffusion Policy, VQ-BeT, QueST, LISA, EXTRACT). The relative improvement over the second-best method increases as demonstrations decrease. For example, on LIBERO-90, BooST achieves relative gains of +140%, +59%, and +41% with 10, 20, and 50 demonstrations, respectively. The paper notes that LISA completely fails in downstream adaptation experiments due to training instability and codebook collapse, providing evidence for the decoupled two-stage design.

2. Cross-embodiment transfer (Q2): BooST successfully transfers skills from the Franka Emika Panda (pretraining) to the UR3 robot (downstream), despite different embodiments and action spaces (joint velocity vs. Cartesian end-effector space). With only five demonstrations per task, BooST achieves the highest success rates across all four real-world manipulation tasks. Low-level methods like VQ-BeT and QueST fail because their skill representations are tied to the pretrained action space.

3. Robustness to dynamic visual distractors (Q3): When pretrained on distractor-augmented LIBERO-90 (with animatable human distractors from AMASS motion sequences), BooST maintains high performance (average 0.90 success rate) compared to latent action pretraining baselines LAPA (0.79) and UniVLA (0.70). The paper explains that "BooST explicitly grounds skill learning in action reconstruction and cross-modal alignment, and further extracts task-aware visual features, collectively forcing the model to focus on task-relevant behavior instead of external noise."

Ablation studies show that both the action pathway (Eact) and task-aware encoder (Etask) are crucial. Removing Eact yields success rates of 0.57, 0.57, 0.67, 0.45 on LIBERO-90, Goal, Object, Spatial respectively, while removing Etask yields 0.25, 0.14, 0.55, 0.30. The full BooST achieves 0.70, 0.67, 0.78, 0.59. Model size experiments show that smaller configurations (29.7M parameters) retain competitive performance (average 0.89) compared to larger models (69.5M: 0.94, 144.5M: 0.93), with downstream models running at approximately 60 Hz.

The paper identifies limitations: because skills are extracted from 2D images, performance can degrade on tasks that demand especially precise motion along the z-axis in camera coordinates and the distilled skill prior is implemented with a non-CLIP-based network, which can face challenges under large viewpoint changes that require stronger 3D understanding. A future direction is to extend BooST with 3D-aware representations, such as incorporating depth estimation models like DepthAnything.

Improvements to AI Systems:

1. Unified What-How Skill Representation with Decoupled Two-Stage Training

• Implement a cross-modal VQ-VAE that jointly encodes semantic intent (via CLIP-based visuo-linguistic features) and motion dynamics (via action trajectories), with alternating optimization of the two pathways.

• Use residual vector quantization (codebook size 16, 2 levels) to create a compact, discrete skill latent that is robust to noise and generalizable across embodiments.

• Decouple pretraining (skill discovery) from downstream adaptation (policy distillation) to avoid codebook collapse and training instability, as seen in LISA’s failure.

• Resulting capability: The system can learn transferable skills from large-scale heterogeneous data (e.g., DROID) and adapt to new tasks with as few as 10 demonstrations, outperforming single-pathway or end-to-end baselines by up to 140% relative improvement.

2. Cross-Embodiment Skill Transfer via Action-Grounded Semantics

• Ground semantic intent in executable action trajectories during pretraining, so the skill representation is not tied to a specific robot’s action space (e.g., joint velocities vs. Cartesian end-effector).

• Add a task-aware encoder (Etask) that extracts task-relevant visual features, filtering out embodiment-specific noise and dynamic distractors.

• Resulting capability: The system can transfer skills from a Franka Emika Panda to a UR3 robot with different action spaces, achieving high success rates (e.g., 5-shot real-world manipulation) where low-level methods like VQ-BeT and QueST fail due to action-space overfitting.

3. Robust Skill Learning Under Dynamic Visual Distractors

• Explicitly optimize for cross-modal alignment (λ3 term) and action reconstruction (λ1, λ2 terms) to force the model to focus on task-relevant behavior rather than static or dynamic visual noise.

• Use a weighted pretraining loss (λ1=3, λ2=1, λ3=0.5) that prioritizes action fidelity while maintaining semantic grounding, making the representation resilient to distractors like human motion sequences.

• Resulting capability: The system maintains high task success (0.90 average) in environments with animatable human distractors, outperforming latent-action pretraining baselines (LAPA: 0.79, UniVLA: 0.70) that lack explicit action grounding.

4. Efficient Downstream Policy Distillation with Lightweight Transformers

• Distill the pretrained skill prior into a small Transformer-based policy (following BAKU architecture) using a two-term objective: prior distillation loss (for skill prediction) and behavior cloning loss (for action generation).

• Use stop-gradient (sg) on the skill latent to prevent interference between prior learning and policy learning.

• Resulting capability: The system can run at 60 Hz on real robots with only 29.7M parameters, retaining competitive performance (0.89 average success) compared to larger models (144.5M: 0.93), enabling real-time deployment on resource-constrained hardware.

5. Improved Few-Shot Generalization via Residual Quantization and Alternating Optimization

• Use residual VQ with multiple quantization levels to capture hierarchical skill granularity (coarse-to-fine), improving representation expressiveness without codebook explosion.

• Alternate between visuo-linguistic and action pathway updates to prevent one modality from dominating, ensuring balanced skill learning.

• Resulting capability: The system achieves superior few-shot performance (e.g., +140% relative gain on LIBERO-90 with 10 demos) and degrades gracefully as demonstration count decreases, unlike baselines that plateau or fail.

6. 3D-Aware Extension for Precision and Viewpoint Robustness

• Integrate depth estimation (e.g., DepthAnything) into the visuo-linguistic pathway to provide 3D spatial cues, addressing limitations in z-axis precision and large viewpoint changes.

• Replace the non-CLIP prior network with a CLIP-based or 3D-aware transformer to better handle perspective shifts.

• Resulting capability: The improved system can perform precise manipulation along the camera z-axis (e.g., vertical insertion tasks) and maintain performance under significant camera viewpoint changes, extending applicability to cluttered or dynamic real-world scenes.

Skill abstraction---the process of learning reusable and temporally extended behaviors---has emerged as a key paradigm for improving sample efficiency and generalization in robot learning. For efficient skill transfer to real robots, learned skills must generalize across tasks and domains, remain robust to visual and dynamic perturbations, and be efficient enough for practical deployment. However, existing methods typically satisfy only a subset of these properties, as they capture either high-level semantic intent (what) or low-level motion dynamics (how). This incomplete skill transfer yields weak priors for policy learning, thereby demanding substantial in-domain data for downstream adaptation. To address these challenges, we introduce BooST, a two-stage framework that explicitly bridges semantics and motions to satisfy all three desiderata. BooST first leverages a cross-modal VQ-VAE to capture both semantic intent and motion dynamics, yielding a unified skill representation. It then distills this representation into a lightweight policy for efficient downstream adaptation to new tasks. Extensive experiments across simulation and real-robot settings demonstrate that BooST achieves superior few-shot adaptation, cross-domain skill transfer, and robustness to dynamic visual distractors, while maintaining a lightweight yet expressive design suitable for real-world deployment.