Pose-Free Feed-Forward 3D Inpainting via Learnable Mask Attention and Support Token Refinement

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Pose-Free Feed-Forward 3D Inpainting via Learnable Mask Attention and Support Token Refinement".

Jane: The gist: FreeInpaint presents a novel feed-forward framework that generates complete and 3D-consistent scenes directly from unposed multi-view images with masked regions, eliminating reliance on precomputed camera poses.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper titled "Pose-Free Feed-Forward three dee Inpainting via Learnable Mask Attention and Support Token Refinement <ref:2610.11857#pg1,Pose-Free Feed-Forward 3D Inpainting via Learnable Mask Attention and Support>." What the authors are trying to do is basically create a way to fill in missing or hidden parts of a three dee scene, but without needing those precomputed camera poses that everyone else usually requires <ref:2610.11857#pg1>.

Jane: That’s right, they present this new feed-forward framework called FreeInpaint. The main idea is that you can generate complete and three dee-consistent scenes directly from a set of unposed multi-view images where some parts are masked out <ref:2610.11857#pg1,complete and 3D-consistent scenes directly from>.

Lu: What makes it novel is that instead of relying on accurately calibrated camera poses first, they extend a three dee foundation model to propagate those masked regions from one reference view to all the other unposed views <ref:2610.11857#pg1,a 3D foundation model to propagate>. It bridges the gap between three dee reconstruction and scene inpainting while keeping the model’s ability to recover geometry and camera parameters intact <ref:2610.11857#pg1,3D reconstruction and scene inpainting while>.

Meng: So, if I understand this right, it means we don't need a heavy pre-processing step like running Structure-from-Motion or COLMAP before we can start inpainting?

Tom: Exactly. The paper points out that existing methods often require that SfM step first, which is time-consuming and brittle when you have sparse views or masked observations. They are trying to make this process faster and more interactive for casual use cases like in-the-wild captures, by eliminating that initial pose estimation bottleneck.

Jane: And they tackle two specific challenges when adapting these feed-forward three dee foundation models to inputs with masks <ref:2610.11857#pg1,feed-forward 3D foundation models to>. First, masked regions can mess up the cross-view correspondence reasoning, which hurts both pose estimation and geometry recovery.

Lu: To handle that, they introduce a Learnable Mask Attention mechanism. This mechanism is designed to keep the spatial anchoring of reliable observations while letting the masked regions gradually soak up useful context in deeper layers. It calculates cross-view attention using this formula: Attention Q K top root d plus B(l) V, where B(l) is a token-level mask bias that starts from an explicit input occlusion mask and gets refined through learnable residuals across layers.

Paper summary: Meng: That sounds like a clever way to stop the masked areas from corrupting the overall scene understanding while still letting them participate in the process. But what about when things are really dark, like severe occlusions, where a single forward pass just doesn't have enough appearance evidence for high-fidelity completion?

Tom: Ah, that’s where their second key challenge comes in. To address that under severe occlusions, they propose a Support Token Refinement strategy. This involves three steps: first, Blind Spot Detection. They compute a visibility score Vi equals P alpha i render times M i over P M i plus epsilon; low opacity means there's not enough support there.

Jane: Then, for the area that needs refinement, they generate a Support View using a diffusion prior conditioned on the current scene context, using Stochastic Differential Editing and IP-Adapter to create a complementary view.

Lu: And finally, they inject those generated support view tokens into the main model in a confidence-weighted manner to help refine those under-observed regions while keeping the original spatial anchor. It's like adding expert guidance when the model feels uncertain.

Meng: From an engineering standpoint, I’m interested in how they handle training this whole system together without needing perfect ground truth poses upfront. The training objectives look pretty detailed on page three and beyond, involving mask generation and depth-guided self-distillation using a frozen copy of DA3 to generate pseudo ground truth depth maps Dprior <ref:2610.11857#pg1>.

Tom: They use a hybrid masking strategy mixing random masks and geometric masks at a ratio of zero point four to zero point six, where the geometric masks are projected from the reference view to the source views using metric depth estimated by the original pre-trained DA3.

Jane: The overall loss function combines a photometric loss, Lrgb, which uses MSE alongside an LPIPS perceptual penalty, with a multi-view depth consistency loss called Ldepth equals norm Dpred times M minus Dprior times M.

Lu: The results they show are quite strong across different datasets like SPIn-NeRF, three hundred sixty-USID, and LLFF. On the SPIn-NeRF dataset alone, FreeInpaint achieves a PSNR of zero point two thousand eight hundred nineteen and an LPIPS of zero point two thousand five hundred twenty-six.

Meng: And the inference speed is impressive for what it does, taking only about two hundredths of a second on a single A6000 GPU for one forward pass. That's fast enough to be useful in many applications.

Paper summary: Tom: Fast is good, but they also highlight some limitations with this paper. They admit that the Support Token Refinement relies on 2D diffusion models to fill those blind spots, so the final three dee quality is limited by how good that 2D prior is <ref:2610.11857#pg1>.

Jane: And they also mention that in extreme cases, like when you only have one or two restricted views of a complex scene, error can accumulate during the support token refinement process. Future work could look at integrating native three dee or multi-view consistent video diffusion priors to make the geometric consistency more stable across arbitrary refinement steps <ref:2610.11857#pg1>.

Lu: They also suggest a persistent spatial memory mechanism for future iterations to maintain long-term geometric consistency across multiple refinement steps, which would solve those accumulation issues. This paper establishes a framework for pose-free three dee scene inpainting, and it shows how you can do this without the traditional heavy preprocessing overhead <ref:2610.11857#pg1>.

Tom: So, what does this mean practically? It means we can get high-quality three dee scenes from just a few images taken casually, bypassing the need for that complex initial calibration step that used to slow everything down <ref:2610.11857#pg1>.

Jane: Basically, it’s about making three dee scene reconstruction and inpainting accessible in scenarios where you don't have perfect camera knowledge from the start <ref:2610.11857#pg1>. This framework, FreeInpaint, shows how to keep a feed-forward model powerful while making it adaptable to real-world data inputs.

Lu: It opens up possibilities for creating interactive applications where users can manipulate three dee scenes directly using unposed inputs without needing a separate SfM pipeline running on the backend <ref:2610.11857#pg1>.

Meng: For practical implementation, the focus will be on making that Support Token Refinement robust enough to handle those truly sparse or heavily occluded scenes we encounter in real-world capture.

Tom: And for the listener, it means future tools for creating three dee environments or editing existing ones won't always need you to start by meticulously setting up a camera rig; they can just give them the images <ref:2610.11857#pg1>.

Jane: That’s what this paper is about: taking a complex task and showing how to solve it using a more direct, feed-forward approach that handles the input variability much better than previous reference-based methods.

Conclusion: Tom: So we've looked at how FreeInpaint tackles three dee scene inpainting without needing those tricky precomputed camera poses that usually slow everything down.

Jane: It really focuses on making this process direct, using unposed images with masked spots to generate a whole, consistent three dee world.

Lu: The authors are doing this by extending an existing three dee foundation model to let it propagate information across those missing parts from one view to all the others.

Meng: So they’re aiming for something fast and adaptable without needing that initial calibration step first.

Tom: Exactly, they’re trying to bypass that heavy pre-processing bottleneck entirely so you can just give them the images and get a three dee scene back quickly.

Jane: And it does this by using two main tricks, learnable mask attention to handle the corrupted views and a support token refinement strategy for when things are really dark.

Lu: The mechanism they use to stop masked areas from messing up pose estimation is called learnable mask attention, which uses a per-token bias that gets refined through layers.

Meng: I see how that would be useful in practice because it keeps the reliable parts anchored while letting the masked regions slowly absorb context.

Tom: And when those areas are totally occluded, they inject support tokens generated by a diffusion model to give the model confidence where it’s blind.

Jane: The overall training involves mixing random and geometric masks and using depth-guided self-distillation to keep the geometry consistent during learning.

Lu: They train it by forcing the predicted depth map to match a pseudo ground truth from a frozen copy of Depth Anything three which anchors the three dee structure.

Meng: On the results side, they show good numbers on datasets like SPIn-NeRF, achieving PSNRs around zero point two eight and inference times under one second on an A6000 GPU.

Tom: Those are solid numbers for pose-free inpainting, but we have to keep an eye on their limitations because they admit the support token refinement still relies on 2D diffusion models for those blind spots <ref:2610.11857#pg1>.

Jane: And they flag that extreme cases, like very few views of a complex scene, can still lead to error accumulation during that refinement step.

Lu: The implication here is that while this is a powerful feed-forward approach, the final three dee quality in the most difficult scenes depends heavily on how good those initial 2D priors are <ref:2610.11857#pg1>.

Meng: It means for real-world use, you might still need a robust way to handle those highly constrained environments if you want perfect geometry.

Tom: So basically, FreeInpaint gives us a path toward quick, pose-free three dee scene generation from just photos, but we’re still working on making it perfectly stable in the worst capture scenarios.

Jane: Next time we talk about this paper, we’ll be looking at how they plan to integrate native three dee diffusion priors to fix those geometric stability issues.