Look Where You Can: Active View Selection for CAD Reconstruction under Occlusion

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Look Where You Can".

Tom: The gist The learned view selection policy outperforms SA-max selection at every budget K ∈

1,..., 5: and the full system surpasses strong external baselines on scene-embedded,

Jane: First, who's behind it and why it matters.

Paper summary: Tom: So we're talking about this paper called "Look Where You Can: Active View Selection for CAD Reconstruction under Occlusion". It’s looking at how to make computer models build three dee designs from pictures when those objects are actually stuck in real-world scenes, like bolted to a wall or jammed into a corner <ref:2610.11954#pg1>.

Jane: Exactly. The big idea here is that current methods assume you can see everything from any angle, which isn't true for real objects. SightCAD introduces a framework that treats choosing the right view as a very important constraint in the reconstruction process.

Lu: It’s treating view feasibility as a first-class constraint, which means the system actively learns where to look based on what’s physically possible in that scene.

Meng: So, instead of just picking random views, this framework uses a learned view selector to pick K feasible views for a vision language model that then generates executable CadQuery code.

Tom: Right, and the scoring for those chosen views comes from how geometrically faithful the resulting three dee solid is after it’s actually rendered from that code <ref:2610.11954#pg1>.

Jane: And they train this whole system together using a joint training paradigm where the view selector and the CadQuery generation vision language model are trained against that geometric fidelity reward.

Lu: The view selector part uses a policy pi psi that maps the full set of calibrated views to logits, which are then used for sampling subsets without replacement from those logits.

Meng: It sounds like they’re using something like a frozen DINOv2 ViT-S/fourteen backbone for the per-view encoding, and then sinusoidal encodings to inject the pose information.

Tom: And that view selector is optimized using Group Relative Policy Optimization, or GRPO, while the generator vision language model learns to exploit those selections through supervised fine-tuning.

Jane: The reward function they use is defined as R(y, O) = IoU, where O is the original object and is the executable program that produces a non-empty solid.

Lu: They mention that this learned selection policy outperforms surface-area maximization, or SA-max, at every budget K from one to five by up to six point four Intersection-over-Union points across all those budgets.

Meng: That's a significant gap closed there, showing that the learned policy is much smarter than just trying to maximize the surface area of what you see.

Tom: And when you look at the final results for this paper, it surpasses strong external baselines on scene-embedded, occluded multi-view renders of DeepCAD and Fusion360 objects with effective mIoU points of twenty-one and seventeen over the best baseline, respectively.

Jane: They also tested it on test-time domain-canonicalized real images from the industrial T-LESS benchmark and both synthetic and real images from the MP6D industrial metal-parts benchmark.

Lu: And in terms of code quality, they produced the highest rate of executable programs compared to other methods, with an invalid code rate of less than one point five percent.

Meng: It's interesting how they break down what makes the selection policy better; it seems to partly recover engineering-drawing projection conventions by prioritizing views that make parameters like extrude depth legible.

Paper summary: Tom: So, when you think about this for someone just driving or cooking, it means we’re getting a tool that can reliably reconstruct a three dee model from messy, real-world photos where the object is clearly partially hidden <ref:2610.11954#pg1>.

Jane: It moves the goal closer to making CAD reconstruction practical for things in factories or cluttered kitchens instead of just pristine digital models.

Lu: The paper also shows that this learned policy supplies more of those specific in-plane views that carry the signal for accurate extruded depth inference than any other view selection policies they considered.

Meng: From an engineering standpoint, that's huge because extrude depth is often a critical dimension, and getting it right from a real photo is tough without knowing exactly where the object sits in space.

Tom: And the numbers they provide are really telling; they lead every baseline on both DeepCAD-test and Fusion360-test, achieving an mIoUeff of fifty-three point five six percent for their setup with K equals four plus GRPO <ref:2610.11954#pg3>.

Jane: They also noted that the T-LESS real evaluation, which feeds real photographs from the wild directly to each method, trails some baselines due to an image distribution gap that canonicalization removes.

Lu: And when they looked at the MP6D industrial metal-parts benchmark, which is acquired on a plain background with no embedding scene, their framework leads every baseline under those specific conditions.

Meng: That suggests that the framework handles the complexity of scene embedding very effectively while still being robust enough for simpler setups.

Tom: So what does this mean for the future? It’s about moving beyond perfect camera setups to building systems that are inherently smart about what information they need to collect when things aren't perfectly visible.

Jane: The focus on active view selection means the AI doesn't just guess; it actively seeks out the most informative views needed for a successful reconstruction.

Lu: This points toward a future where vision models don’t just describe scenes but intelligently guide the observation process to get the right data for a task.

Meng: For practical use, this means fewer failed reconstructions when you're trying to reverse-engineer an existing physical part from photos, whether it's on an assembly line or in a workshop.

Tom: That’s where we end with this look at "Look Where You Can: Active View Selection for CAD Reconstruction under Occlusion". It shows how learning to choose the right camera angles can bridge the gap between digital design and messy reality.

Jane: It really emphasizes that for real-world applications, knowing what to see is just as important as having a good model of what you see.

Lu: The implication is a smarter interaction between vision and generative models where feasibility is baked into the learning objective from the start.

Meng: We're looking at systems that can be more reliable when working with imperfect visual data, which is exactly what most of us deal with daily in industrial settings.

Conclusion: Tom: So we’re wrapping up this paper, "Look Where You Can: Active View Selection for CAD Reconstruction under Occlusion." Basically, they figured out how to make AI look at a messy real-world object and intelligently decide which pictures it actually needs to see.

Jane: Right. They’re moving away from the old idea that you just show it everything, and instead using a learned system to pick the best views for building that three dee model.

Lu: The authors are focusing on treating view feasibility as a first-class constraint in their framework. It’s about making sure the AI doesn't waste time looking at corners or walls that don't give useful information.

Meng: So, when we think about this practically, it means for someone trying to reverse-engineer a part from photos, the computer doesn't just guess; it actively chooses the views that actually help build something solid.

Lalam: From a cultural standpoint, this suggests that vision models can become much more useful in industrial settings where physical objects are often cluttered and partially hidden. It makes the AI’s output more reliable for real-world tasks.

Tom: And they did some heavy lifting on the math there too. They showed their view selection policy actually beats a simple method called SA-max by over six point intersection-over-union across all those view budgets, K equals one to five.

Jane: That’s a big number for how much better the selection mechanism is compared to just picking the biggest surface area they can see.

Lu: What’s really interesting is that their policy seems to pick views that make engineering drawing conventions more obvious, especially when it comes to figuring out things like extrude depth.

Meng: That's where I think the real-world impact is. If the AI prioritizes those specific views, it gets a better handle on crucial dimensions like how deep a part is extruded from its sketch.

Tom: Exactly. It’s not just about getting a pretty picture; it’s about getting the right data points to actually make an executable program that works perfectly.

Jane: So, in short, this paper shows that active view selection makes CAD reconstruction much smarter when objects aren't perfectly visible.

Lalam: It means the next generation of vision-to-CAD models can be far more robust when dealing with the kind of messy data we see every day in manufacturing or inspection.

Tom: It really shifts the focus from just generating a model to intelligently gathering the right information for that model. **(Music swells slightly)**

Jane: Next up, we’re going to look at how they tested this against some very real-world industrial metal parts data and see if those gains hold up when things get truly complex.