HARMONI: Aligning Human and Scene Priors for Multi-View 4D Reconstruction

arXiv:2603.12789 · cs.CV · Submitted 2026-03-13 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "HARMONI: Aligning Human and Scene Priors for Multi-View 4D Reconstruction".

Jane: HARMONI is a unified framework designed to jointly reconstruct cameras, scene point clouds, and human meshes from multi-person multi-view videos in a single pass without requiring external modules or preprocessing.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Moving on, the paper introduces the concept of CHROMM as this unified framework, and I want to talk about what that title really implies for us listeners. It’s all about aligning human priors with scene priors during multi-view reconstruction.

Jane: Exactly; it’s not just reconstructing a scene or just reconstructing a human; it’s achieving alignment between those two things simultaneously across multiple camera angles, which is where the real challenge lies in this research.

Lu: The authors are tackling the problem of extending existing methods that usually require either monocular inputs or significant computational overhead for multi-view extensions, and they present a solution that integrates these priors into one network architecture.

Meng: I’m thinking about the practical implications here; if you cut out all those external modules like 2D keypoint detectors, you simplify the entire system design considerably for any real-world application.

Lalam: For culture and applications, this means that AI systems won't need separate specialized tools for every task; they can just start reconstructing complex scenes and human interactions directly from raw video feeds.

The paper's summary: Tom: Now let’s look at the core summary of "HARMONI: Aligning Human and Scene Priors for Multi-View 4D Reconstruction." Essentially, they describe how their approach jointly estimates cameras, scene point clouds, and human meshes in a single pass.

Jane: That means they are tackling the entire pipeline—camera parameters, three dee geometry of the scene, and the three dee shape of the humans—all at the same time without needing separate stages for each component.

Lu: The paper explains that they begin with dual-feature encoding: flattening video frames into a sequence and extracting two representations: one for the scene captured by Pi3X, and another specifically for human representation extracted from Multi-HMR.

Meng: So, it’s using these two distinct features differently—feeding scene tokens into the Pi3X decoder for scene reconstruction, while human features go straight to the human head reconstruction head. That dual-feature strategy is key there.

Lalam: From a cultural perspective, this ability to handle multi-person, multi-view video in one go means that AI can capture complex social scenes with much richer contextual understanding than current methods allow.

The paper's improvements: Tom: What really stands out in the summary is the specific improvements they introduce to make this work even better, particularly concerning scale and fusion. They propose a metric decoder within Pi3X to estimate a global scale factor s in R.

Jane: That’s important because it directly addresses a known issue where approximate metric-scale scene predictions don't perfectly match the SMPL meshes, so they introduce this "scale adjustment module" using the head-pelvis length ratio for refinement.

Lu: They also tackle multi-view fusion by decomposing the human representation into view-invariant and view-dependent components, fusing invariant attributes like shape parameters beta and canonical pose theta, while transforming view-dependent attributes into a shared world coordinate system.

Meng: From a practical standpoint, that scale adjustment module sounds like a critical fix; if the scene reconstruction is off by an unknown factor of scale, all subsequent measurements become unreliable.

Lalam: The way they handle multi-person association using explicit geometric cues—three dee positions and poses—instead of relying on appearance features, seems like a very robust way to ensure that we can reliably track multiple individuals in a crowd.

Conclusion: Tom: So, wrapping up this discussion on "HARMONI: Aligning Human and Scene Priors for Multi-View 4D Reconstruction," the main point is that by integrating strong geometric and human priors into one architecture, they achieve competitive performance in global human motion estimation and multi-view pose estimation while running over eight times faster than prior optimization-based methods.

Jane: That speedup combined with the ability to do everything in a single pass really makes this work very compelling for practical research and application development right now. It shows that we can achieve high accuracy without needing those heavy, time-consuming optimization loops that used to be standard.

Lu: The paper confirms that relying only on Pi3X scale prediction causes performance drops because of the scale gap, so enabling the "scale adjustment module" is essential for consistent performance across different scene scales.

Meng: It’s good to hear about these explicit geometric cues for association; that moves us away from unreliable appearance-based re-identification methods when dealing with scenarios where people look very similar.

Lalam: This work really pushes the potential of AI to handle complex, real-world video data holistically, suggesting future systems can process massive amounts of visual information with unprecedented speed and accuracy in a single operation.

Seoul National University · NAVER Cloud

cs.CV

Submitted: 2026-03-13

Updated: 2026-09-30

Project page: https://nstar1125.github.io/chromm

Importance score: 83/100

The gist: HARMONI is a unified framework designed to jointly reconstruct cameras, scene point clouds, and human meshes from multi-person multi-view videos in a single pass without requiring external modules or

Key concepts

HARMONI: A unified framework designed to jointly reconstruct cameras, scene point clouds, and human meshes from multi-person multi-view videos in a single pass without needing external modules or preprocessing.
Dual-feature encoding: The approach involves flattening video frames into a sequence and extracting two representations: one for the scene captured by Pi3X, and another specifically for human representation extracted from Multi-HMR. These features are used differently in the reconstruction process.
Scale adjustment module: A proposed metric decoder within Pi3X that estimates a global scale factor 's' in R. This module refines approximate metric-scale scene predictions by using the head-pelvis length ratio to address performance drops caused by scale gaps.

Terminology

Accuracy — 12× in this paper · explained in 1 episode
Computer Vision and Pattern Recognition — 11× in this paper
Ablation — 10× in this paper · explained in 1 episode
Unified Framework — 10× in this paper · explained in 5 episodes
Alignment — 8× in this paper · explained in 1 episode
Consistency — 7× in this paper · explained in 3 episodes
Foundation Model — 7× in this paper · explained in 3 episodes
Foundation Models — 5× in this paper · explained in 7 episodes
Human Mesh Recovery — 5× in this paper
Point Clouds — 5× in this paper · explained in 2 episodes
Robustness — 5× in this paper · explained in 5 episodes
Transformer — 4× in this paper · explained in 2 episodes
Benchmark — 3× in this paper · explained in 2 episodes
Optimal Transport — 3× in this paper · explained in 5 episodes
Two-stage training — 3× in this paper · explained in 2 episodes
Ablation Studies — 2× in this paper · explained in 1 episode
AdamW Optimizer — 2× in this paper · explained in 1 episode
Computational overhead — 2× in this paper · explained in 2 episodes
Embeddings — 2× in this paper · explained in 2 episodes
Late Fusion — 2× in this paper · explained in 2 episodes
Contextual Understanding — 1× in this paper · explained in 2 episodes
Core — 1× in this paper · explained in 2 episodes
Decoupling — 1× in this paper · explained in 2 episodes
Downstream Tasks — 1× in this paper · explained in 2 episodes
Generalization — 1× in this paper · explained in 13 episodes
Grounding — 1× in this paper · explained in 3 episodes
Multimodal Dataset — 1× in this paper · explained in 2 episodes

Summary

HARMONI is a unified framework designed to jointly reconstruct cameras, scene point clouds, and human meshes from multi-person multi-view videos in a single pass without requiring external modules or preprocessing. This approach addresses the limitations of existing methods that typically rely on monocular inputs or incur significant computational overhead for multi-view extensions. By integrating strong geometric and human priors into a single trainable neural network architecture, HARMONI aims to achieve competitive performance in global human motion estimation and multi-view pose estimation while running over 8× faster than prior optimization-based approaches.

Model Architecture and Dual-Feature Encoding

HARMONI builds upon Pi3X [51], a 3D foundation model capable of predicting approximate metric-scale geometry, and adopts SMPLX [35] for human modeling. The pipeline begins with dual-feature encoding: flattening the multi-view video frames into a single sequence and extracting two representations: (i) a scene-wise feature Fscene n, captured by the Pi3X encoder to capture global 3D geometry, and (ii) a human-wise feature Fhuman n, extracted by an additional encoder from Multi-HMR [3] specifically trained for human representation. The architecture processes these features differently: scene tokens are fed into the Pi3X decoder for scene reconstruction, while human features bypass the Pi3X decoder and go directly to the human reconstruction head.

Scene Reconstruction and Scale Adjustment

The model reconstructs camera parameters (πv t) and local 3D point maps (Pv t) from decoded scene tokens. A key innovation is the introduction of a metric decoder within Pi3X, which estimates a global scale factor s ∈ R. This predicted scene scale s is applied to local point maps and camera translations across all frames, enabling approximately metric-scale scene reconstruction. To solve the scale discrepancy between the approximate metric-scale scene prediction and the SMPL meshes, HARMONI proposes a scale adjustment module that utilizes the head-pelvis length for refinement. This is calculated by computing a ratio of head-pelvis lengths observed in the image versus those of projected SMPLs, leading to an adjusted metric scale s∗ = r · s.

Multi-View Fusion Strategy

To aggregate per-view human estimates into a coherent global representation, HARMONI introduces a test-time optimization-free multi-view fusion strategy. This strategy decomposes the predicted human representation into view-invariant and view-dependent components. View-invariant attributes, such as the shape parameter β and canonical pose parameters θ, are fused by computing the mean of per-view predictions for individuals corresponding to the same identity. View-dependent attributes, like root rotation R and 3D head translation τ, are transformed into a shared world coordinate system using estimated camera extrinsics. The global 3D head position is computed via multi-view ray triangulation to enforce multi-view consistency.

Multi-Person Association Method

Accurate cross-view person identities are established through a geometry-based multi-person association method that leverages explicit geometric cues, such as estimated 3D positions and human poses, rather than relying on external reidentification (ReID) modules. The process involves two steps:

Tracking: Humans in each view are tracked using pairwise L2 distances between consecutive human tokens, with outlier estimates filtered by comparing the estimated global 3D human joints between consecutive time steps to suppress erroneous estimations.
Association: Cross-view association is performed by constructing a view connectivity graph and computing a matching cost C(a, b) that combines a position term (from global 3D joints) and a pose term (from canonical-space joints). This cost benefits from the robustness of 3D positional consistency while allowing pose similarity to resolve edge cases.

Training Regimes and Ablation Insights

HARMONI is trained in two stages: Stage 1 focuses on learning human reconstruction by freezing the backbone encoders and training SMPL decoders with geometric, parametric, and detection losses. Stage 2 unfreezes only the pelvis detection MLPs to train them using pelvis loss, reprojection losses, and Chamfer distance loss to enforce geometric consistency. Ablation studies confirm that relying solely on Pi3X scale prediction leads to performance drops due to the scale gap; enabling the scale adjustment module is crucial for consistent performance across diverse scene scales. Furthermore, ablations show that explicitly averaging view-invariant parameters yields the highest performance compared to implicit token max-pooling.

Experimental Performance

Experiments on datasets like EMDB and RICH demonstrate that HARMONI achieves competitive performance in both global human motion estimation and multi-view human pose estimation tasks without external modules or optimization. The model achieves a "competitive performance in both global human motion estimation and multi-view human pose estimation tasks, while providing over 8× speedup compared to other optimization-based multi-view approaches.

Improvements for AI systems

Here are the specific improvements that can be made to AI systems by implementing CHROMM, along with what these improved systems will be capable of:

The system will achieve high-fidelity, unified 3D reconstruction from multi-person, multi-view video inputs in a single pass without needing external modules (like 2D keypoint detectors) or complex preprocessing.
The system will jointly estimate the camera poses, the scene point cloud geometry, and the 3D meshes of multiple humans simultaneously from raw video feeds.
The system will accurately reconstruct human motion and pose parameters in a global world coordinate system, even when individuals are viewed from different perspectives (multi-view consistency).
The system will perform robust multi-person association across different views by leveraging explicit geometric cues (3D positions and poses) rather than relying on potentially ambiguous appearance features, leading to reliable tracking of multiple people in a scene.
The system will handle significant scale discrepancies between the reconstructed scene geometry and the human meshes by dynamically computing a scale adjustment ratio based on the head-to-pelvis length ratio across multiple views, ensuring accurate metric-scale reconstruction.
The improved AI system will be significantly faster than prior optimization-based multi-view approaches, achieving over 8x speedup in inference time while maintaining competitive reconstruction accuracy (e.g., lower W-MPJPE and RTE).
The system will provide enhanced performance on complex tasks such as global human motion estimation and multi-view pose estimation, outperforming existing feed-forward models like JOSH3R, UniSH, and Human3R in both monocular and multi-view settings.
The system will be robust to challenging real-world conditions, including frame cropping (using coarse predictions when the pelvis is outside boundaries) and partial occlusions (using the pelvis as a stable reference for scale adjustment).