ModPack: Extensible Teleoperation Interface for Bimanual Mobile Manipulation

ModPack introduces a modular and extensible teleoperation system designed to support diverse robot embodiments and task requirements within a unified framework, addressing the limitations of existing systems that are often tailored to specific hardware. The core concept is a self-contained wearable “backpack” that integrates onboard computation, power, communication, and data storage.

How it works

ModPack is built on a shared interface where the backpack acts as a common substrate, decoupling core system infrastructure from task- or robot-specific functionalities. This foundation supports plug-and-play capability modules that allow operators to flexibly add or remove extensions based on task needs. Key capabilities enabled by these modules include:

1. Cross-robot joint-level control through swappable leader arms, enabling adaptation to diverse robot arm kinematics and supporting precise joint-level control across different robot platforms.

2. Mobility, achieved by tracking the operator’s base motion via an iPhone mounted on ModPack, which enables mobile manipulation for tasks that require large workspace coverage.

3. Active perception, where ModPack streams real-time robot head-camera observations while allowing the operator to control the camera pose, enabling demonstrations of behaviors such as object search, occlusion handling, and viewpoint selection.

4. Haptic feedback provided by leveraging force feedback from the robot and active motors on leader arms to enable safer and more precise teleoperation of contact-rich tasks.

System Architecture

The core system is organized around a portable backpack that provides the necessary hardware for modular extensions. The internal volume is organized into five removable shelves housing the onboard compute, power, and optional accessories. The software stack abstracts communication using a lightweight message queue, which decouples the core backpack stack from embodiment-specific implementations. This allows the same system to support different robots and arbitrary subsets of extended modules.

Extensible Modules

The system is extensible through three primary modules: Leader Arms, Mobile Base, and Active Perception.

Leader Arms:

These modules consist of 6-DoF ARX5 or 7-DoF leader arms constructed to be kinematically equivalent to its corresponding follower arm, enabling direct joint-space teleoperation. To enhance stability, active gravity compensation is implemented, where the gravity torque vector is computed using the Orocos Kinematics and Dynamics Library (KDL), followed by filtering using two cascaded exponential moving average (EMA) filters to obtain the final commanded torque. Haptic feedback is integrated by mapping net wrench from force/torque sensors to joint-space torques via the Jacobian transpose, resulting in a final control torque that is the superposition of gravity compensation and haptic feedback.

Mobile Base:

The operator’s SE(2) pose is tracked using an iPhone running a modified WebXR session. An egocentric mapping procedure couples the robot’s base translation directly to the operator’s displacement within their local frame, ensuring that when the operator walks forward, the robot moves forward in its own local frame. A sequence of offsets is applied to recover the user’s true center of rotation, accounting for physical displacement between the iPhone and the operator. This tracked pose is then mapped into world coordinates to issue absolute target setpoints to the robot base controller.

Active Perception:

This module utilizes an Apple Vision Pro running a custom visionOS app to stream real-time robot head-camera observations. The system tracks the operator’s head and body motion, and compensates for base motion before sending adjusted robot head commands. This decoupling enables operators to naturally walk, look around, and control the arms during data collection.

Policy Learning

For policy training, a standard transformer-based Diffusion Policy model is trained for each task using the collected teleoperation demonstrations. The input modalities vary by robot: for the custom mobile robot, the policy takes RGB-D observations from the head camera and RGB observations from the left and right wrist cameras. For the RB-Y1m robot, it takes RGB observations from two head cameras and the left and right wrist cameras, and optionally left and right arm joint torques. All visual inputs are encoded using a CLIP-pretrained ViT, with depth images repeated across three channels before encoding. Torque observations are encoded with sequential CausalConv layers.

Experimental Evaluation

ModPack was evaluated through real-world bimanual mobile manipulation experiments across two distinct robot platforms: the customized mobile robot with ARX5 arms and holonomic base for Cloth Placement, and an RB-Y1m robot for Box Transfer with Haptic Feedback. The results demonstrated the flexibility of the design, showing that policies trained on this data achieve strong deployment performance, validating ModPack's utility as a practical data collection interface for robot learning. For instance, in the Cloth Placement task, policies conditioned on All Cam + Torque achieved higher success rates compared to those using only head or wrist cameras.

As a fastidious and diligent researcher, I have analyzed the ModPack paper. The core innovation lies in creating a highly modular, unified teleoperation framework that decouples operator input from robot embodiment and task requirements.

Here are the specific improvements to AI systems derived from this work, detailing what these improved systems can achieve:

1. The system can support the development of a generalized Teleoperation Policy Learning Framework capable of rapidly adapting to novel robot hardware and task domains without requiring complete retraining.

2. This framework allows for the creation of high-fidelity imitation learning policies (e.g., Diffusion Policies) by leveraging heterogeneous, multi-modal sensory inputs (RGB, RGB-D, proprioception, force/torque signals).

3. The system enables the creation of robust Cross-Modal Perception Policies that can dynamically select the optimal visual input modality—such as switching between head-camera only views for long-horizon search and full camera/torque views for contact-rich manipulation—based on real-time task context (as demonstrated by the success of 'Head Cam' vs. 'All Cam' policies).

4. The system allows for the development of Haptic-Conditioned Control Policies where arm joint torques are explicitly conditioned on force/torque feedback signals, enabling the robot to learn precise contact-rich behaviors like stable grasping and controlled object placement (as evidenced by the superior performance of 'All Cam + Torque' policies in Box Transfer).

5. The system facilitates Simultaneous Multi-Objective Control Policies that explicitly disentangle operator motion into component-wise commands (arm, base, perception), allowing the learned policy to manage complex, coupled dynamics—such as mobile manipulation while simultaneously tracking a dynamic visual target—without operator ambiguity.

6. The framework can be used to generate Active Perception Policies that automatically optimize viewpoint selection during data collection or execution based on task progress (e.g., automatically shifting from a search pattern to a detailed inspection view when an object is detected).

7. The system enables the creation of Ergonomic Control Policies that incorporate gravity compensation and haptic feedback into the learned control loop, resulting in policies that are inherently safer and more robust for bimanual manipulation under physical load constraints.

This improved AI system can perform:

1. Perform complex, bimanual mobile manipulation tasks (e.g., Cloth Placement, Box Transfer) across a wide variety of robot embodiments (different leader arms/kinematics).

2. Execute dexterous contact-rich manipulation by accurately controlling force application and object placement using real-time haptic feedback.

3. Navigate large workspaces while maintaining precise manipulation through coordinated base motion and arm control.

4. Learn generalized skills by ingesting diverse, multi-modal human demonstrations (visual, proprioceptive, force) and producing high-performance policies that are robust to variations in robot hardware and environmental conditions.

Existing teleoperation systems are often tailored to specific robot hardware and task domains, limiting their scalability and adaptability. We present ModPack, a modular and extensible teleoperation system designed to support diverse robot embodiments and task requirements within a unified framework. At the core of ModPack is a self-contained wearable "backpack" that integrates onboard computation, power, communication, and data storage. Built on top of this shared interface, the system supports plug-and-play capability modules including joint-level teleoperation with haptic feedback, mobile manipulation, and active perception. Experiments across two distinct robot platforms and real-world mobile manipulation tasks demonstrate that ModPack provides a flexible and reusable framework for data collection and policy learning and collects higher quality data than a robot-free alternative. To support future research, we open-source the complete hardware design and software stack. Project website: https://modpack-robotics.github.io/