From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generation

arXiv:2605.08712 · cs.CV · Submitted 2026-05-09 · Read on arXiv

cs.CV

Submitted: 2026-05-09

Updated: 2026-09-25

Comments: NeurIPS 2026: https://arlo0o.github.io/KVLR-project/

Journal ref: NeurIPS 2026

Code: https://github.com/facebookresearch/slowfast

License: http://creativecommons.org/licenses/by/4.0/

The gist: Action-conditioned surgical video generation is a critical yet highly challenging problem for robotic surgery.

Terminology

Abstract

Action-conditioned surgical video generation is a critical yet highly challenging problem for robotic surgery. The core difficulty is that low-dimensional control vectors must precisely govern complex image-space evolution. In this work, we propose a kinematic-to-visual lifting paradigm that converts articulated kinematics into a unified set of five image-aligned control modalities. Building on this representation, we introduce a hierarchically routed visual control framework that selectively activates the most relevant control modalities and motion scales. Instead of uniformly applying all control signals, our model performs hierarchical routing to dynamically allocate conditioning capacity. We further design kinematic-prior-guided routing loss functions to ensure physically meaningful, temporally stable, and efficient expert utilization. To improve efficiency, we propose a budgeted training and inference scheme that leverages routing-induced sparsity. By selectively discarding low-significance control pathways during training and execution, our approach enables adaptive computation that is complementary to standard distillation. We additionally construct a new benchmark with curated articulated annotations, obtained through human-in-the-loop semantic labeling and differentiable pose tracking, providing realistic supervision for action-conditioned surgical video generation. Extensive experiments demonstrate that our method consistently improves action faithfulness, visual fidelity, and cross-domain generalization over diverse baselines. Moreover, our efficient variant achieves substantial reductions in latency while maintaining strong control accuracy.

Sources

Related papers