BGM2Pose: Active 3D Human Pose Estimation with Non-Stationary Sounds
cs.CV, cs.AI, cs.SD, eess.AS
Submitted: 2025-03-01
Updated: 2026-09-13
Comments: Accepted by the IEEE Open Journal of Signal Processing, Vol. 7, 2026. Project page: https://yutoshibata07.github.io/bgm2pose-project-page/
Project page: https://yutoshibata07.github.io/bgm2pose-project-page
License: http://creativecommons.org/licenses/by/4.0/
The gist: We propose BGM2Pose, a non-invasive 3D human pose estimation method using arbitrary music (e.g., background music) as active sensing signals.
Terminology
Abstract
We propose BGM2Pose, a non-invasive 3D human pose estimation method using arbitrary music (e.g., background music) as active sensing signals. Unlike existing approaches that significantly limit practicality by employing intrusive chirp signals within the audible range, our method utilizes natural music that causes minimal discomfort to humans. Estimating human poses from standard music presents significant challenges. In contrast to sound sources specifically designed for measurement, regular music varies in both volume and pitch. These dynamic changes in signals caused by music are inevitably mixed with alterations in the sound field resulting from human motion, making it hard to extract reliable cues for pose estimation. To address these challenges, BGM2Pose introduces a Contrastive Pose Extraction Module that employs contrastive learning and hard negative sampling to eliminate musical components from the recorded data, isolating the pose information. Additionally, we propose a Frequency-wise Attention Module that enables the model to focus on subtle acoustic variations attributable to human movement by dynamically computing attention across frequency bands. Experiments suggest that our method outperforms the existing methods, demonstrating substantial potential for real-world applications. Project page: https://yutoshibata07.github.io/bgm2pose-project-page/
Sources
- Markerless Motion Capture and Biomechanical Analysis Pipeline
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Score-Based Generative Modeling through Stochastic Differential Equations
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Acoustic-based 3D Human Pose Estimation Robust to Human Position
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models