HiPHI: A Large-Scale Benchmark for High-Precision Human Motion and Object-Interaction
cs.RO, cs.AI
Submitted: 2026-08-17
Updated: 2026-09-08
Comments: Accepted at CoRL 2026. Project page: https://noitom-robotics.github.io/hiphi/
Code: https://github.com/NVLabs/ProtoMotions
Project page: https://noitom-robotics.github.io/hiphi
License: http://creativecommons.org/licenses/by/4.0/
The gist: Humanoid intelligence requires learning over an extremely diverse space of whole-body motions and physically grounded interactions.
Terminology
Abstract
Humanoid intelligence requires learning over an extremely diverse space of whole-body motions and physically grounded interactions. However, existing embodied datasets remain fundamentally limited: internet-scale video data lack precise physical states and interaction grounding, while laboratory motion datasets provide high fidelity but only narrow behavioral coverage. This mismatch creates a critical bottleneck for scalable humanoid policy learning. We present HiPHI, a 600+ hour scale high-fidelity whole-body human motion dataset designed to systematically maximize coverage of the human motion and interaction manifold. HiPHI is theoretically guided by FrameNet, a linguistic framework organizing human primitives. Created using an optical motion capture pipeline, HiPHI provides sub-millimeter spatial marker tracking accuracy for full-body human motion and mesh-level object trajectories. We further introduce a benchmark suite evaluating motion-space diversity, interaction grounding, object consistency, and physical AI applications. Our analyses demonstrate that HiPHI significantly expands motion coverage compared to existing motion datasets while maintaining high-fidelity interaction quality, and establishes a scalable data foundation for training, evaluating, and generalizing humanoid policies in real-world embodied tasks, where similar extensions are also applicable to motion prior models in computer graphics. Project page: https://noitom-robotics.github.io/hiphi/
Sources
- Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots
- DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
- EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
- H2R: A Human-to-Robot Data Augmentation for Robot Pre-training from Videos
- Motion-X++: A Large-Scale Multimodal 3D Whole-body Human Motion Dataset
- BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion
- SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control
- OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interaction
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies
- Fine-tuning is Not Enough: A Parallel Framework for Collaborative Imitation and Reinforcement Learning in End-to-end Autonomous Driving