HiReFF: High-Resolution Feedforward Human Reconstruction from Uncalibrated Sparse-View Video

arXiv:2606.29333 · cs.CV · Submitted 2026-06-28 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "HiReFF: High-Resolution Feedforward Human Reconstruction from Uncalibrated Sparse-View Video".

Jane: HiReFF is a feed-forward method designed for high-resolution (2K) 360° human video reconstruction from uncalibrated, sparse-view videos,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we’re talking about the paper "HiReFF: High-Resolution Feedforward Human Reconstruction from Uncalibrated Sparse-View Video" and what it actually proposes to do. Basically, the core idea is taking four uncalibrated ninety-degree spaced videos and turning them into a high-resolution three dee Gaussian Splatting representation of a human in a streaming way at three point zero one frames per second while hitting 2K resolution.

Jane: That’s a big leap from what we usually see, isn't it? The paper lays out that the goal is to map those high-resolution multi-view videos to a dynamic three dee Gaussian Splatting representation by estimating both the Gaussians and temporally smooth per-view camera parameters.

Lu: What really stands out in their summary is how they tackle two major issues right away: scale mismatch and foreground reconstruction problems, which they say are key challenges in this setup.

Meng: I’m curious about the technical details of those two specific challenges; understanding exactly how they handle them will tell us a lot about the actual feasibility for deployment.

Lalam: From my perspective, their summary shows a clear path toward more realistic digital humans because it’s tackling the physical reconstruction problem directly rather than just synthesizing images frame by frame.

The paper's summary: Tom: Let's talk about the specifics of what HiReFF actually does, beyond just the high-level goal. The summary explains that they are reconstructing a three hundred sixty-degree human in a streaming fashion at three point zero one FPS using only four input views, and they achieve 2K resolution with only thirty-four percent additional VRAM during training compared to lower resolutions.

Jane: That efficiency metric is pretty compelling; getting 2K output without ballooning the memory usage is crucial for any practical application, especially when we think about running these models on consumer hardware.

Lu: The summary highlights that they decompose the problem into foreground three dee Gaussian reconstruction and then efficient high-resolution synthesis, which is a clever way to manage complexity in a feed-forward structure.

Meng: Decomposing the problem sounds like good engineering practice; it suggests they're not trying to solve everything at once but breaking it down into manageable parts that contribute to the final output.

Lalam: It’s about making the reconstruction process robust enough that you don't have to worry about every single detail individually, which makes the final result more reliable for use in things like interactive virtual environments.

The paper's improvements: Tom: Now, let’s get into the actual methods they propose to solve those problems. They introduce three core components: Scale-synchronized Camera Calibration, Gaussian-wise Foreground Masking, and High-resolution Side-tuning. These are the mechanisms they use to overcome those initial hurdles we discussed.

Jane: The scale adjustment part is interesting because they acknowledge that just having four ninety-degree views isn't enough for supervision without ground truth camera parameters, so they propose dynamically adjusting parameters during training and using indirect supervision for extra views.

Lu: And then there’s the Gaussian-wise Foreground Masking, which involves a mask head to modulate the Gaussian parameters based on predicted foreground probabilities, specifically because direct masking hurts camera parameter accuracy in these wide-baseline settings.

Meng: That masking component sounds tricky to implement reliably; ensuring that modulating the Gaussians doesn't corrupt the underlying camera estimation is a real technical tightrope walk for an engineer.

Lalam: I think that mechanism shows a deep understanding of the interplay between geometry and appearance, which is what’s needed to make these reconstructed humans look physically plausible and not just like blurry shapes.

Conclusion: Tom: So, wrapping up the discussion on "HiReFF: High-Resolution Feedforward Human Reconstruction from Uncalibrated Sparse-View Video," it seems the authors have successfully put together a framework that handles scale ambiguity, foreground masking issues, and achieves 2K resolution streaming at a reasonable speed.

Jane: Exactly. The main implication is that we can achieve high-fidelity volumetric video reconstruction for humans using only sparse inputs without needing perfect camera calibration or massive computational resources upfront.

Lu: For the future, I think the real potential lies in how this feed-forward approach can be adapted to handle even more complex scenarios, perhaps integrating features from other models like those used in ProCompNav to improve query handling.

Meng: Practically speaking, if we can deploy this on a single GPU for streaming at three point zero one FPS, it opens up possibilities for real-time interactive holographic applications that were previously too resource-intensive.

Lalam: This work points toward a future where creating detailed, consistent digital human avatars in AR and VR becomes much more standard because the reconstruction pipeline itself is significantly more robust and scalable.

Yiming Jiang, Hanzhang Tu, Wenfeng Song, Siyou Lin, Li Anliang, Shuai Li

State Key Laboratory of Virtual Reality Technology and Systems, Beihang University

cs.CV

Submitted: 2026-06-28

Updated: 2026-06-28

DOI: 10.1007/978-3-032-37350-2_16

Project page: https://iridescentjiang.github.io/HiReFF

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 87/100

The gist: HiReFF is a feed-forward method designed for high-resolution (2K) 360° human video reconstruction from uncalibrated, sparse-view videos, addressing the critical need for temporal consistency and

Key concepts

HiReFF
A feed-forward method designed for high-resolution (2K) 360° human video reconstruction from uncalibrated, sparse-view videos. It maps these inputs to a dynamic three dee Gaussian Splatting representation in a streaming manner.
Gaussian Splatting
A representation used by HiReFF to store the reconstructed human. It involves estimating both the Gaussians and temporally smooth per-view camera parameters to achieve high-resolution output.
Scale-synchronized Camera Calibration
A core component that dynamically adjusts camera parameters during training. This addresses scale ambiguity, acknowledging that four ninety-degree views are insufficient without ground truth camera parameters.
Gaussian-wise Foreground Masking
A mechanism involving a mask head to modulate Gaussian parameters based on predicted foreground probabilities. This helps manage complexity while trying to maintain accurate camera estimation.

Terminology

Summary

HiReFF is a feed-forward method designed for high-resolution (2K) 360° human video reconstruction from uncalibrated, sparse-view videos, addressing the critical need for temporal consistency and computational efficiency in applications like holographic communication and AR/VR. It tackles the limitations of existing methods—which are often restricted to low resolution or require calibrated cameras—by decomposing the problem into foreground 3D Gaussian reconstruction and efficient high-resolution synthesis. This framework enables streaming reconstruction at 3.01 FPS on a single GPU while achieving 2K resolution with only a modest computational overhead compared to prior models.

Problem Setup and Goal

The primary goal of HiReFF is to reconstruct a high-resolution (HR) 3D Gaussian Splatting (3DGS) volumetric video from four uncalibrated, 90° spaced input views. The model performs the mapping from high-resolution multi-view videos to a dynamic 3DGS representation by estimating:

  1. A set of G anisotropic 3D Gaussians, parameterized by position, opacity, rotation, scale, and spherical harmonic coefficients.

  2. Temporally smooth per-view camera parameters for each view.

The overall model performs the mapping from high-resolution multi-view videos to a high-resolution dynamic 3DGS representation:

fθ: 4i=1 → (µt g, Θt g) G g=1 ∪ (p ti) 4i=1 T t=1, enabling high-resolution novel view synthesis at H HR × W HR resolution and arbitrary temporal intervals.

Key Components of the Framework

HiReFF decomposes its approach into three core components to achieve its objectives:

Scale-synchronized Camera Calibration:

This component addresses the scale mismatch hindering additional view supervision. Since input views are separated by 90°, supervision from only these views is insufficient, requiring additional novel viewpoint supervision which necessitates ground-truth camera parameters. To resolve this, HiReFF proposes Scale-synchronized Camera Calibration to dynamically adjust camera parameters for extra supervision views during training and employs indirect supervision of the camera head. Furthermore, it addresses scale ambiguity by scaling true camera parameters based on an average of the translation vectors from multiple views:

(3) The scale adjustment is computed as: bar s = (1 / (V-1)) sum i=2 V (t i / t i), and hat t'i = hat t i / bar s for i = 2,..., V.

Gaussian-wise Foreground Masking:

This component addresses the challenge where directly masking input images for foreground-only reconstruction severely degrades camera parameter accuracy in widely-spaced (90°) sparse-view settings. To ensure only the foreground human region is reconstructed, HiReFF introduces a mask head to modulate Gaussian parameters. This process is defined as:

(2) (Θt g, µt g) g=1 H = F m((Θt g, µt g) g=1 G) ⊙ (Θt g, µt g) G,

where ⊙ denotes Gaussian-wise masking based on the predicted foreground probabilities.

High-resolution Side-tuning:

To achieve computationally efficient high-resolution reconstruction without increasing the AA Transformer input resolution beyond 518×518, HiReFF employs a side-tuning strategy inspired by prior work. This approach augments the Gaussian head with supplementary features to enable high-resolution output. Specifically, it maintains the input of the AA Transformer at 518×518 while providing complementary high-resolution information to the Gaussian head through:

(5) Θ t = F D(F a(I t) ⊕ U(F G mid(I t))),

where F a is an MLP modified by EdgeNeXt [31] and U denotes upsampling to high resolution. This ensures that high-resolution rendering quality depends primarily on the high-dimensional Gaussian attributes Θ g output by the Gaussian head, rather than on position µ g or camera position p i.

Training and Loss Functions

The training objective combines several losses:

Rendering Loss:

A rendering loss is used to supervise image synthesis.

Improvements for AI systems

Here are specific improvements that can be made to existing AI systems by adopting or integrating the concepts from HiReFF, and what those improved systems could achieve:

  1. Improve 3D Human Reconstruction Fidelity in Uncalibrated Environments:

  2. Enable High-Resolution (2K) Streaming Volumetric Video Synthesis from Sparse Views:

  3. Enhance Robustness to Large Viewpoint Baselines (e.g., 90° spacing):

  4. Facilitate Real-time, Computationally Efficient Novel View Rendering for AR/VR:

  5. What the Improved AI System Can Do (Specific Capabilities):

Feature Original Limitation Addressed Specific Capability of HiReFF-based System Impact on Application

:---:---:---:---

Limited to low-resolution (0.5K) single-frame synthesis. (Page 1, Abstract) Achieves native 2K resolution output during streaming reconstruction at a stable frame rate (3.01 FPS). (Page 2) Enables high-fidelity, photorealistic holographic communication and AR/VR experiences without sacrificing visual quality due to low resolution.

Reliance on calibrated cameras or per-scene optimization for sparse views. (Page 1, Intro) Performs full 360° human reconstruction from only four uncalibrated, 90°-spaced videos using feed-forward methods. (Page 1, Fig. 1) Allows deployment of volumetric video reconstruction in real-world scenarios where cameras are not precisely calibrated (e.g., mobile AR/VR tracking).

Scale ambiguity hinders multi-view supervision and camera alignment. (Page 3) Introduces Scale-synchronized Camera Calibration to resolve metric scale ambiguity by dynamically adjusting camera parameters during training and using indirect supervision. (Page 3, Sec 3.2) Enables accurate, metric-scale human model reconstruction where the output size is meaningful in real-world units (e.g., meters), crucial for physics simulation and interaction in VR/AR.

Foreground masking compromises camera parameter accuracy in wide baselines (90°). (Page 3) Implements Gaussian-wise Foreground Masking which modulates Gaussian parameters based on predicted foreground probabilities, ensuring clean reconstruction without degrading camera estimation. (Page 3, Sec 3.2) Produces cleaner, more accurate reconstructions of human subjects by effectively separating foreground geometry from background noise in sparse-view setups.

High computational cost limits feed-forward models to low resolutions (518x518). (Page 4, Section 3.3) Utilizes High-resolution Side-tuning which augments the Gaussian head with supplementary image features while keeping the backbone at a lower resolution (0.5K), achieving 2K rendering efficiency. (Page 6, Sec 3.3) Drastically reduces VRAM and computational overhead during both training and inference compared to high-resolution direct models, making real-time streaming feasible on consumer GPUs (e.g., RTX 4090).

Difficulty in maintaining temporal consistency across complex clothing details. (Page 12, Fig. 5) Demonstrates excellent temporal consistency for both the human body and intricate garment patterns during reconstruction sequences. (Page 12, Fig. 5) Ensures smooth, high-quality motion and deformation of clothing when viewing the reconstructed human model from different angles over time, vital for realistic avatar interaction.

Need for novel view synthesis compatible with feed-forward backbones. (Page 5) Introduces a sidetune adaptation strategy that injects image-derived features into the mid-level of the Gaussian head, enabling fast and efficient High-Resolution Novel View Synthesis (HRNVS). (Page 5, Sec 3.3) Allows for photorealistic rendering from any arbitrary novel viewpoint using the reconstructed 3D Gaussians in a feed-forward manner, overcoming limitations of traditional HRNVS methods like SRGS.

Sources

Related papers