RoboAssist: Interactive Human-Humanoid Planning for Long-Horizon Surgical Assistance

RoboAssist presents an agent-based framework for interactive human–humanoid planning designed to enable long-horizon surgical assistance by coordinating robot actions with evolving human activities while ensuring safety across planning and execution. The core contribution is an asymmetric dual-track representation that separates partially observed human process states from executable robot task sequences, allowing the system to maintain online dependency alignment and perform targeted residual replanning when workflow requests change.

The gist: RoboAssist is an agent-based framework for interactive human–humanoid planning that integrates workflow reasoning, task coordination, and cross-layer safety.

Core Framework and Representation

RoboAssist is built around an asymmetric dual-track representation that separates partially observed human process states from executable robot tasks, linking them through evidence-gated dependencies. This design is crucial because it avoids mixing non-controllable process estimates with robot-controllable actions. The system maintains two distinct tracks: the Human Process Track (Ut) and the Robot Task Track (Rt).

The Human Process Track, denoted as Ut, defines human progress through nodes where each node includes a human state or action description, associated entities, and an assistance need, linked through observation-updated task memory (Mt). The Agent estimates procedural progress from observations and predicts subsequent actions. A condition is only confirmed when the criterion τi is satisfied by admissible observational evidence or explicit human confirmation; missing evidence leaves it unconfirmed.

The Robot Task Track, Rt, represents a sequence of executable robot task nodes, where each node includes the bound skill (kj), invocation parameters (θj), execution status (σj), and its precondition (pj) and expected effect (ej). The Agent generates this sequence from confirmed requirements and anticipated assistance needs using workflow knowledge.

Interactive Planning and Online Alignment

The framework achieves online task alignment by linking the two tracks through a Gating Map, Zt = (Ut, Rt, ϕt), where ϕt(j) indexes the human-process nodes on which task vj depends. This map provides traceable human–robot dependencies for consistency checking and residual replanning.

At each decision cycle, new observations update the state memory (Mt). The Agent then revises Ut+1 and checks the remaining robot tasks for changes in requirements or dependencies. For an unfinished task vj, gate satisfaction is evaluated by calculating gj (Ut+1) = i∈ϕt(j) τi(Ut+1). If a gate is unsatisfied, the task waits; if a physical precondition pj (s) is unmet, it is checked at dispatch.

Cross-Track Consistency and Residual Replanning

Residual replanning is triggered when new evidence invalidates a requirement or human-process dependency, or when a required precondition can no longer be established. The divergence point k∗ is defined as the earliest unfinished task whose requirement or human-process dependency is invalidated, or whose physical precondition can no longer be established by the retained prefix R<k∗t. The prefix R<k∗t is preserved, and only the affected suffix is regenerated to ensure new tasks remain compatible with the expected effects of preserved tasks.

Cross-Layer Safety Architecture

RoboAssist employs a three-layer safety design that coordinates protection across planning, handover, and runtime execution:

1. Preventive surgeon-aware navigation regulation: This layer regulates locomotion by commanding vcmd t = (0, dt ≤ 0, α(dt) vnom t, dt > 0), where α(dt) is a clearance-dependent velocity scaling factor that approaches unity far from the protected region and issues a zero-velocity command at boundary contact or entry.

2. Reactive interaction regulation for VLA handover: This layer separates task-oriented motion generation from interaction safety regulation. A VLM-based monitor evaluates the relative human–robot configuration, and if unexpected proximity is detected, it interrupts nominal behavior to issue an avoidance response.

3. Fail-safe whole-body runtime supervision: An independent supervisor continuously monitors proprioceptive feedback using the anomaly flag et = eF t ∨ eV t ∨ eO t (force, joint velocity, and persistent oscillation detectors). If the latched emergency-stop state is activated (lt), it terminates the current skill, inhibits subsequent motion commands, and has the highest authority over other layers.

Experimental Evaluation

The framework was implemented on a Unitree G1 humanoid robot in long-horizon, multi-stage simulated surgical assistance scenarios involving multimodal interaction, instrument handling, medical material transport, navigation, and safe human–robot handover. The experiments compare RoboAssist against a Scripted FSM and a Reactive agent. In the request-order adaptation benchmark (where delivery order changes), RoboAssist achieved an 81.25% success rate compared to 0% for the scripted FSM, while significantly reducing mean replanning latency from 3.62 to 1.

Here are the specific improvements for AI systems based on RoboAssist, and what these improved systems can achieve:

1. Improve long-horizon task coordination in complex human-robot environments (e.g., surgery). The system will transition from simple command execution to a sophisticated agent capable of reasoning about evolving surgical workflows, anticipating surgeon needs, and managing multi-stage procedures requiring sequential dependencies (like picking up an instrument, moving it to a handover table, and then handing it over).

2. Implement robust online task adaptation via residual replanning. The improved system will maintain an asymmetric dual-track representation separating uncertain human process states from executable robot tasks. When human requests change or observations invalidate prior assumptions, the AI will only replan the affected suffix of the task sequence, drastically reducing planning latency and computational overhead compared to full-plan regeneration.

3. Integrate a cross-layer safety architecture for surgical autonomy. The system will employ three distinct safety layers:

4. Preventive navigation regulation (decelerating as it approaches a surgeon's protected operating region).

5. Reactive interaction regulation during handovers (interrupting nominal motion if unexpected proximity or human movement is detected).

6. Fail-safe whole-body runtime supervision (monitoring joint velocity, force feedback, and oscillations to trigger an emergency stop upon detecting anomalies).

These improvements result in an AI system capable of performing:

4. Predictive instrument provisioning and material transport within surgical workspaces while maintaining a surgeon-aware safety margin.

5. Seamless adaptation to dynamic workflow changes requested by the surgeon (e.g., changing the order of required deliveries) with near-instantaneous planning response times, ensuring task continuity without long stalls or complete re-planning cycles.

6. Safe, reliable physical interaction during close-range handovers and high-precision manipulation by dynamically adjusting motion based on real-time human proximity and physical anomaly detection.