MiDAS: A Multimodal Data Acquisition System and Dataset for Robot-Assisted Minimally Invasive Surgery

Summary

This paper introduces MiDAS, an open-source, platform-agnostic system for multimodal data acquisition in robot-assisted minimally invasive surgery (RMIS). The system enables time-synchronized, non-invasive data collection across surgical robotic platforms without requiring proprietary robot interfaces. MiDAS integrates electromagnetic hand tracking, RGB-D hand tracking, foot pedal sensing, and surgical video capture. The authors validated MiDAS on the open-source Raven-II and the clinical da Vinci Xi platforms by collecting multimodal datasets of peg transfer and hernia repair suturing tasks performed by surgical residents. Correlation analysis and downstream gesture recognition experiments were conducted, demonstrating that external hand and foot sensing closely approximated internal robot kinematics, and non-invasive motion signals achieved gesture recognition performance comparable to robot kinematics. The system and annotated datasets are publicly released, including the first multimodal dataset capturing hernia repair suturing on high-fidelity simulation models.

The paper states: We introduce MiDAS, an open-source, platform-agnostic system enabling time-synchronized, non-invasive multimodal data acquisition across surgical robotic platforms. The methods section notes: MiDAS integrates electromagnetic and RGB-D hand tracking, foot pedal sensing, and surgical video capturing without requiring proprietary robot interfaces. The results indicate: External hand and foot sensing closely approximated internal robot kinematics, and non-invasive motion signals achieved gesture recognition performance comparable to robot kinematics. The conclusions state: MiDAS enables reproducible multimodal RMIS data collection and is released with annotated datasets, including the first multimodal dataset capturing hernia repair suturing on high-fidelity simulation models.

The paper's contributions are fourfold: (1) presenting MiDAS as an open-source system that can be non-invasively integrated with standard teleoperated robotic surgery systems to collect fully synchronized data in real-time; (2) proposing new alternative modalities to proprietary robot kinematics, including surgeon hand and foot motion kinematics captured using electromagnetic hand tracking sensors, depth cameras, and force sensors, demonstrating they closely approximate surgeon and robot kinematics and can be effectively used for tasks such as activity recognition; (3) evaluating MiDAS by collecting data from two RMIS tasks—standard Peg Transfer on Raven-II and a novel multimodal dataset of hernia repair suturing on realistic KindHeart tissue models using the da Vinci Xi, performed during a training bootcamp at the University of Virginia Hospital—and identifying effective modality combinations for multimodal gesture recognition by benchmarking state-of-the-art Transformer and CNN models; (4) publicly releasing both the data collection system and the resulting multimodal, multi-platform datasets with gesture annotations.

The MiDAS architecture employs a client-server design: "A central server coordinates multiple client processes that stream heterogeneous data (e.g., kinematics, video) as shown in Figure 10 at different rates, implemented via Python multiprocessing workers to enable concurrent, real-time ingestion and buffering." The server provides a GUI for entering study metadata, selecting modalities, and monitoring system status. All streams are stamped on arrival at the MiDAS server with a Unix epoch timestamp, providing a unified, server-side time base. The estimated cost for a single surgeon-console MiDAS setup was approximately 8,000 USD, including the NDI trakSTAR electromagnetic tracking unit (4,850), Blackmagic SDI recorders (400), a ZED Mini RGB-D camera (400), a custom pedal sensing system (150), and a laptop workstation (2,000). The data recording usage rate is approximately 0.4 GB/minute when capturing at 30 Hz with 1080p stereoscopic video.

For electromagnetic hand tracking, the system employs the NDI trakSTAR device with four miniature 6-DoF sensors: Two sensors are mounted on each MTM control at the thumb and middle-finger contact pads (Figure 1), preserving the console hardware while measuring fine manipulator motions. The system logs 3D position and orientation for each sensor at 270 Hz. Raw sensor data are mapped from the electromagnetic tracker frame to robot reference frames via a calibrated rigid-body transformation augmented with a learned residual modeled using a Multi-Layer Perceptron (MLP). The MLP models have 2 hidden layers with 16 neurons, use ReLU activation, and are trained with root mean squared error objective, L2 weight decay, 50% dropout, Adam optimizer with learning rate 0.005, over 50 epochs.

For RGB-D hand tracking, a ZED Mini stereo camera is mounted above the surgeon console, providing synchronized 720p color video and dense depth maps at 30 Hz. Hand keypoints are extracted using Mediapipe's hand landmark detection library, and 3D positions are computed using the aligned RGB and depth frames. The keypoints are transformed into the MTM base frame using either a rigid body transformation from hand-eye calibration or an end-to-end MLP transformation. The paper notes: Limited camera field-of-view (FoV) and occlusions cause intermittent keypoint detection gaps. We fill gaps of at most one second using cubic spline interpolation and longer gaps using forward-filling.

For video capture, the system uses Open Broadcaster Software (OBS) as a unified capture layer. On da Vinci Xi, stereoscopic SDI output is captured using Blackmagic SDI recorders, while on Raven-II, stereo video is recorded using a ZED camera. OBS records synchronized, high-definition stereo streams at 30 Hz, controlled programmatically via the OBS WebSocket API.

The Pedal Sensing System (PSS) captures surgeon foot pedal interactions using an Arduino-class microcontroller and thin-film force-sensitive resistors (FSRs) affixed directly to console pedal surfaces. PSS supports up to nine pedals and records both binary pedal states and timestamped analog signals at 30 Hz. Each FSR is mounted in a voltage-divider with a known series resistor, and a brief per-pedal calibration establishes voltage thresholds for robust binary actuation detection, further refined through data-driven optimization.

Two datasets were collected. The Raven-II dataset consists of 15 Peg Transfer trials, resulting in 36 minutes of data with gestures annotated using the DESK taxonomy, involving 7 gesture classes (Approach peg, Align & grasp, Lift peg, Transfer peg – Get together, Transfer peg – Exchange, Approach pole, Align & place) with 345 total samples. Ground-truth kinematic data including MTM and PSM position, orientation, velocity, grasper angle, and clutch pedal presses were available from the open-source platform. The da Vinci Xi dataset was collected during a robotic surgery training bootcamp at the University of Virginia hospital, with 40 participants (mostly 2nd and 3rd year surgical residents) performing 7 trials of inguinal hernia repair and 10 trials of ventral hernia repair, totaling 3.5 hours of annotated suturing segments. The suturing taxonomy includes 8 gestures: Orient Needle, Target Needle, Push Needle Through Tissue, Pull Needle out of Tissue, Reach Suture, Pull Suture, Make C Loop, and Square Knot and Cinch, with 1,724 total samples across 212 minutes. The taxonomy was developed in collaboration with an expert robotic surgeon and iteratively refined and validated.

Survey responses from bootcamp participants indicated strong acceptance: "92.3% of participants reporting overall satisfaction with the training experience and 86.1% rating the KindHeart tissue models as realistic. Importantly, 80% of participants disagreed or strongly disagreed that the sensors negatively affected their psychomotor performance."

For validation, the authors performed cross-modal correlation analysis comparing EmHT and HandKP data against ground-truth internal MTM and PSM kinematics on Raven-II. For positional trajectories, EmHT-MTM and EmHT-PSM mean Cosine Similarity (CoS) exceeded 0.8 across all axes, with Normalized Root Mean Square Error (NRMSE) below 24.4%. HandKP achieved mean CoS exceeding 0.7 for all axes with superior 0.88-0.89 Z-axis alignment. For orientation, EmHT-MTM achieved CoS of 0.90 for Yaw, though Pitch and Roll showed moderate alignment (0.61 and 0.65). EmHT-PSM orientation alignment was lower, with CoS values ranging from 0.32 (Roll) to 0.44 (Pitch), attributed to the complex control loop and Raven-II's controller errors. For grasper angle, EmHT sensors achieved moderate agreement with both MTM and PSM grasper angles (IoU ≈ 0.53-0.54, Accuracy ≈ 0.80-0.81).

For pedal sensing validation, the PSS was compared against ground-truth pedal data from both platforms. On Raven-II, PSS achieved F1 of 0.85, precision of 0.78, recall of 0.95, IoU of 0.74, and lag of 166.67 ms. On da Vinci Xi, PSS achieved F1 of 0.78, precision of 0.69, recall of 0.86, IoU of 0.67, and lag of 133.33 ms. The paper notes: "The lower precision (0.69) arises primarily from the intrinsic high-sensitivity design of the pedal sensing system, which is biased toward detecting all true presses (i.e., favoring recall) and therefore produces occasional false positives."

For downstream gesture recognition, the authors benchmarked two state-of-the-art temporal models, MTRSAP and MS-TCN++, on both datasets. On Raven-II Peg Transfer, MTM/PSM kinematics achieved F1-scores of 0.87-0.88. Notably, using non-invasive EmHT achieves comparable performance to MTM/PSM, with MTRSAP model gaining an F1-score of 0.86 and accuracy of 0.87, closely matching baseline MTM/PSM models. HandKP performed suboptimally (F1-score of 0.4) due to suboptimal pose approximation and missed hand detections. On da Vinci Xi Suturing, EmHT performed comparably to vision-only models and improved accuracy when fused with them: MTRSAP EmHT+Image (DINOv2) achieving Acc of 0.71 and F1-score of 0.70, outperforming each single modality. Image-only models underperformed due to limited data, with DINOv2 features outperforming ResNet-50.

The paper also reports ablation studies. For feature ablation on Raven-II, incorporating clutch state information led to notable performance gains for EmHT (F1 improved from 0.77 to 0.86). Removing grasper state caused macro-F1 to drop significantly from 0.78 to 0.47 in MTM performance and 0.75 to 0.58 in PSM performance. Incorporating velocity consistently yielded substantial gains across kinematic modalities. For visual feature adaptation, pretraining ResNet-50 on the in-domain DESK dataset yielded 15-30% relative improvement, and replacing ResNet-50 with DINOv2 features provided further gains.

The paper concludes: "By releasing the MiDAS system, multimodal datasets, and baseline benchmarks, we aim to support reproducible, cross-platform research, reduce reliance on closed robotic systems, and accelerate progress in multimodal learning for surgical activity understanding." The work was partially supported by research awards from the National Science Foundation (CNS-2146295) and the Commonwealth Cyber Initiative. The surgical bootcamp and associated data collection were approved by the University of Virginia Institutional Review Board under protocol IRB-SBS 6724, and all participants provided informed consent.

Based on the paper, here are specific improvements I can make to AI systems and what the improved systems can do:

Improvement: I will implement a cross-modal attention fusion mechanism that combines electromagnetic hand tracking (EmHT) signals with vision features (DINOv2), as demonstrated by the MTRSAP model achieving 0.71 accuracy and 0.70 F1-score on da Vinci Xi suturing—outperforming single-modality baselines by 2-4% absolute.

Capability: The improved system can recognize 8 distinct suturing gestures (needle orientation, targeting, pushing through tissue, pulling, suture reaching, pulling suture, C-loop formation, square knot cinching) in real-time from non-invasive sensors, even when video quality degrades due to occlusion or lens contamination.

Background: Robot-assisted minimally invasive surgery (RMIS) research increasingly relies on multimodal data, yet access to proprietary robot telemetry remains a major barrier. We introduce MiDAS, an open-source, platform-agnostic system enabling time-synchronized, non-invasive multimodal data acquisition across surgical robotic platforms. Methods: MiDAS integrates electromagnetic and RGB-D hand tracking, foot pedal sensing, and surgical video capturing without requiring proprietary robot interfaces. We validated MiDAS on the open-source Raven-II and the clinical da Vinci Xi by collecting multimodal datasets of peg transfer and hernia repair suturing tasks performed by surgical residents. Correlation analysis and downstream gesture recognition experiments were conducted. Results: External hand and foot sensing closely approximated internal robot kinematics and non-invasive motion signals achieved gesture recognition performance comparable to proprietary telemetry. Conclusion: MiDAS enables reproducible multimodal RMIS data collection and is released with annotated datasets, including the first multimodal dataset capturing hernia repair suturing on high-fidelity simulation models.