Surgical WAM: A World-Action Model for Data-Efficient Surgical Robot Learning

Surgical WAM: A World-Action Model for Data-Efficient Surgical Robot Learning

Summary

This paper introduces the Surgical World-Action Model (Surgical WAM), a unified generative model for surgical robot learning that jointly predicts future endoscopic observations and executable surgical robot action chunks. The central research question is: under a fixed budget of action-labeled demonstrations, does action-free video pretraining improve closed-loop surgical manipulation? The authors answer this affirmatively through a two-stage training recipe.

Motivation and Problem Setting

The paper addresses the scarcity of action-labeled surgical robot data. Collecting teleoperated dVRK trajectories with synchronized kinematics is costly, requiring access to surgical hardware, specialized environments, expert operators, and task-specific setup. Existing surgical manipulation datasets contain only hundreds to a few thousand demonstrations, far fewer than the millions used in general-purpose robot learning. In contrast, endoscopic video is comparatively inexpensive and abundant. The authors note that existing surgical world models use video primarily for simulation or policy evaluation, and rarely translate the learned dynamics into closed-loop control.

Model Architecture and Training

Surgical WAM is built on Cosmos Policy, a diffusion-transformer realization of a World-Action Model. At each control step, the model takes the current endoscopic observation and dVRK proprioceptive state, and jointly predicts a future video sequence together with an action chunk. The training is decoupled into two stages:

1. Stage 1 (Action-free video pretraining): The world-model component is trained on unlabeled surgical video using only the video loss, learning general dynamics and spatiotemporal representations from abundant unlabeled videos. The action and state slots are excluded from both input and supervision.

2. Stage 2 (Action-labeled fine-tuning): Starting from the video-pretrained initialization, the full model is fine-tuned on action-labeled demonstrations using the joint objective that couples action prediction with future-video prediction. This allows the model to transfer video-derived dynamics representations while learning to predict executable surgical actions.

At deployment, the model acts as a closed-loop, receding-horizon controller that executes a short prefix of each predicted action chunk and replans from the resulting observation.

Main Results

The model is evaluated on four SurRoL benchmark tasks: Needle Pick, Peg Transfer, Needle Regrasp, and BiPeg Transfer, covering unimanual, bimanual, and contact-rich behaviors. Key results from Table 1:

• Surgical WAM w/ PT achieves the best success rates: 99% on Needle Pick, 86% on Peg Transfer, 62% on Needle Regrasp, and 64% on BiPeg Transfer.

• Surgical WAM w/o PT achieves 96%, 66%, 50%, and 42% respectively.

• Video pretraining improves average success from 63.5% to 77.8%, an absolute gain of 14.3 points.

• The largest gain is on PegTransfer: an absolute improvement of 20 percentage points (from 66% to 86%).

• The improvements are most pronounced on the contact-rich and bimanual problems.

The paper also compares against prior policies: BET, DEX, ALOHA, Diffusion Policy, and π0.5. Notably, generalist policies transfer poorly—ALOHA drops to 14% on PegTransfer, Diffusion Policy to 2%, and π0.5 obtains only 8% on NeedlePick and 12% on NeedleRegrasp. The authors note that language-conditioned action decoding alone does not capture the fine-grained visual dynamics required in the surgical domain.

Ablation Studies

1. Fine-tuning steps (Table 2): The video-pretrained model peaks at 86% after only 80k steps, whereas the model without pretraining requires 160k steps to reach its best of 79%. The pretrained initialization attains a higher peak with half the optimization budget. At longer horizons (120k, 160k steps), the pretrained model degrades, attributed to overfitting to the limited demonstrations that gradually erodes the visual-dynamics prior.

2. Execution horizon (Table 3): The pretrained model is robust across execution horizons He ∈ 1, 2, 4, 8, maintaining at least 70% success, whereas the model without pretraining is more sensitive and peaks only at He = 4 (66%). The default He = 4 is adopted.

Real Surgical Video Validation

The authors apply the identical two-stage protocol to demonstrations from the JIGSAWS dataset, a corpus of dVRK operation recordings captured on real robotic hardware. They observe the same qualitative trend as in simulation: video-pretrained initializations yield stronger downstream policies than a from-scratch baseline. This confirms the advantage of action-free video pretraining is not an artifact of the simulator's rendering or dynamics, but persists under the visual complexity and manipulation difficulty of real surgical scenes.

Contributions and Conclusions

The paper makes three main contributions: (1) developing Surgical WAM, to our knowledge the first world-action model applied to surgical robot learning; (2) answering the central question—under a fixed action-labeled budget, video pretraining improves closed-loop surgical manipulation, with gains coming entirely from the video pretraining, at no extra cost in action supervision; and (3) characterizing when the benefit is realized, showing the pretrained model peaks with half the fine-tuning steps, is robust across execution horizons, and gains most on contact-rich and bimanual tasks.

The authors conclude that abundant unlabeled surgical video is an effective substitute for costly action labels, and that scaling action-free pretraining is a promising direction for data-efficient surgical robot learning. The bottleneck shifts from collecting expensive teleoperated demonstrations to curating video that the surgical domain already produces at scale.

Based on the paper, here are the specific improvements I can make to AI systems and what the improved systems can do:

Improvement 1: Two-stage training with action-free video pretraining

• I can pretrain the world-model component on unlabeled video (learning spatiotemporal dynamics) before fine-tuning on action-labeled data.

• The improved system can achieve 14.3 percentage points higher average success rate (77.8% vs 63.5%) on surgical manipulation tasks, with the largest gain of 20 points on contact-rich tasks like Peg Transfer.

Improvement 2: Joint prediction of future video and action chunks

• I can couple action prediction with future-observation prediction in a single generative model, using the video prediction as a self-supervised auxiliary signal during fine-tuning.

• The improved system can perform closed-loop, receding-horizon control, executing short action prefixes and replanning from new observations, leading to robust performance across execution horizons (maintaining ≥70% success for He ∈ 1,2,4,8).

Improvement 3: Decoupled state/action slots during pretraining

• I can exclude action and state slots from both input and supervision during the video pretraining stage, forcing the model to learn purely visual dynamics.

• The improved system can transfer these visual dynamics to action learning, achieving peak performance with half the fine-tuning steps (80k vs 160k) and avoiding overfitting to limited demonstrations.

Improvement 4: Task-agnostic video pretraining for data-scarce domains

• I can apply this recipe to any domain with abundant unlabeled video but scarce action labels, not just surgery.

• The improved system can generalize to real-world settings (validated on JIGSAWS) and outperform generalist policies (e.g., π0.5, Diffusion Policy) that fail on fine-grained visual dynamics, by leveraging domain-specific video priors.

Improvement 5: Robustness to execution horizon and fine-tuning budget

• I can design the model to be less sensitive to hyperparameters like execution horizon and fine-tuning steps, reducing the need for extensive tuning.

• The improved system can maintain high success rates even with shorter execution horizons (He=1) and shorter fine-tuning budgets, making it more practical for deployment in real surgical settings where compute and data are limited.

Learning reliable surgical manipulation policies is bottlenecked by the scarcity of action-labeled demonstrations: teleoperated surgical robot (e.g., dVRK) trajectories with synchronized kinematics are costly to collect, while surgical tasks demand precise contact handling, long-horizon reasoning, and bimanual coordination. Endoscopic video is comparatively inexpensive and abundant relative to synchronized video--kinematics trajectories, and a natural way to exploit it is to learn world models of surgical scenes. However, existing surgical world models use video primarily for simulation or policy evaluation, and rarely translate the learned dynamics into closed-loop control. This gap raises our central question: under a fixed budget of action-labeled demonstrations, does action-free video pretraining improve closed-loop surgical manipulation? To answer it, we introduce the Surgical World-Action Model (Surgical WAM), a unified generative model built on Cosmos Policy that jointly predicts future endoscopic observations and executable surgical robot action chunks. Surgical WAM first learns surgical visual dynamics from action-free video and is then fine-tuned on the fixed action-labeled budget; at deployment, it acts as a closed-loop, receding-horizon controller that executes a short prefix of each predicted action chunk and replans from the resulting observation. On a suite of four simulated surgical manipulation tasks, video pretraining improves the average success rate from 63.5% to 77.8%, including an absolute gain of 20 percentage points on PegTransfer, with the largest improvements on contact-rich and bimanual tasks. These results demonstrate that action-free video provides transferable visual dynamics priors for learning surgical robot control with limited action supervision, positioning data-efficient video pretraining as a practical path toward scaling up surgical robot learning.