MVMD: A Multi-View Approach for Enhanced Mirror Detection

arXiv:2608.07559 · cs.CV, cs.LG · Submitted 2026-08-02 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "MVMD: A Multi-View Approach for Enhanced Mirror Detection".

Jane: The paper was written by Yidan Shen, Yu Wen, Chen Zhang, Xin Fu and Renjie Hu from University of Houston.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Jane: We also have Lu with us today — senior AI researcher at Tsinghua.

Tom: We also have Meng with us today — lead engineer at a mysterious AI startup.

Jane: We also have Lalam with us today — the in-house Large Language Model.

Tom: Alright, let's get started.

Title and Authors: Tom: Welcome back to the arXiv Review, everyone! I'm Tom, and as always, I'm joined by my co-host Jane. Today we're looking at a paper that's got me genuinely excited: "MVMD: A Multi-View Approach for Enhanced Mirror Detection."

Jane: And I'm Jane! Tom, I have to say, when I first read that title, I thought, "Mirror detection? Isn't that just... looking at a mirror?" But this is so much more interesting than that. The authors are from the University of Houston — Yidan Shen, Yu Wen, Chen Zhang, Xin Fu, and Renjie Hu.

Tom: Right, and the core problem they're tackling is that mirrors completely break three dee reconstruction. You know how when you take photos of a room with a mirror, the reflection looks like a whole other room? Computers get fooled into thinking that reflection is real space.

Jane: Exactly! And that's a huge deal for things like virtual reality, autonomous navigation, even architecture. If a robot or a VR system thinks there's a doorway where there's actually just a mirror, it's going to bump into a wall or create a completely wrong virtual space.

Tom: So the paper's big move is using multiple camera angles — multi-view images — to spot mirrors. The idea is that when you move the camera, the stuff inside the mirror shifts differently than the stuff outside it. That difference is the clue.

Jane: And that's why the title says "Multi-View." They're not just looking at one photo; they're comparing several shots of the same scene from slightly different positions. It's like how you can tell a painting of a window from a real window by moving your head side to side — the painting doesn't change perspective the way a real window does.

Tom: That's a perfect analogy, Jane. And the authors built a whole new dataset for this, which we'll get into later. But first, let me just say — the fact that they're from the University of Houston and they're publishing on arXiv, this is the kind of fundamental computer vision work that could make three dee reconstruction actually reliable in real-world spaces.

Jane: And that's what I love about this paper. It's not just an academic exercise. It's solving a problem that affects so many applications we're starting to rely on. We'll dig into the technical details next, but stick around — this one's a gem.

Tom: Absolutely. Next up, we're going to break down the abstract and the core challenge they're addressing. Don't go anywhere!

Summary and Core Challenge: Jane: Welcome back! We're still on "MVMD: A Multi-View Approach for Enhanced Mirror Detection," and Tom, I want to get into the meat of the abstract because there's a really important number in there.

Tom: Oh, you mean the eleven point one percent improvement in IoU? That's the Intersection over Union metric — basically how well the predicted mirror mask overlaps with the actual mirror in the image. That's a massive jump.

Jane: Massive! And they also improved accuracy by two point six percent. But the more interesting part to me is the problem statement. They say mirrors create "phantom objects" and "distorted geometries" in three dee reconstruction. That's such a vivid way to put it.

Tom: It really is. And the reason this happens is that algorithms like NeRF, three dee Gaussian Splatting, and even classic multi-view stereo like COLMAP — they all assume that light travels in straight lines from a surface to the camera. Mirrors break that assumption completely.

Jane: Right! When the camera sees a reflection, it's seeing light that bounced off the mirror surface. The algorithm thinks that light came from behind the mirror. So it reconstructs a fake room, a fake hallway, fake objects that don't actually exist in the scene.

Tom: And that's why the authors emphasize that existing single-image mirror detection methods aren't enough. They only see one perspective, so they can't tell the difference between a mirror and, say, a window or a dark doorway. But with multiple views, you can actually track how reflections move differently from real objects.

Jane: Exactly. And here's the clever part — they designed the network to use three images. The first and second images have a small angle between them, and the first and third have a larger angle. That way, the network can compare small shifts and large shifts in the reflection.

Tom: That's the "Inter-Views Block" we'll talk about in a second. But the key takeaway from the abstract is that they're not just improving on existing methods — they're opening up a whole new input format for mirror detection. Multi-view input, not single image, not video.

Jane: And they built a dataset to go with it. We'll get to that in the next segment, but let me just say — the fact that they created ninety-eight scenes with three thousand one hundred eighty-one images just for this task shows how seriously they take the problem.

Tom: It does. And that dataset is going to be a gift to the research community. Next up, we're going to talk about the three main blocks of their network architecture. Stay with us!

Improvements and Methodology: Tom: Welcome back to our discussion of "MVMD: A Multi-View Approach for Enhanced Mirror Detection." Jane, we've talked about the problem and the dataset. Now let's get into the actual architecture, because that's where the real innovation is.

Jane: Yes! And I love how they structured it around three key observations about mirrors. First, reflections change as you move the camera. Second, objects inside the mirror correspond to objects outside the mirror. Third, mirrors have distinct edges. Each observation maps to a specific block in the network.

Tom: So the first block is the Inter-Views Block. It takes the high-level features from all three images and applies cross-attention between image one and image two, and between image one and image three. This lets the network focus on what changed between views.

Jane: And that's brilliant because it's not just looking at differences — it's learning which differences are caused by mirror reflections versus which are caused by the camera actually moving. A wall doesn't change its relationship to the camera the way a reflection does.

Tom: Right. Then there's the Intra-View Block. This one's really clever. It takes the target image, flips it horizontally, and compares the mirror region to the flipped non-mirror region. Because a mirror reflection is literally a flipped version of the real scene, the network learns to match objects inside the mirror to their real-world counterparts.

Jane: That's such a smart trick. It's like saying, "Hey, if there's a lamp on the left side of the room, and I see a lamp on the right side of that shiny rectangle, that rectangle is probably a mirror."

Tom: Exactly. And finally, the Refinement Block. This one sharpens the edges of the predicted mirror mask. It uses two parallel convolution layers — one that looks at fine local details and one that looks at the broader surrounding context. By subtracting those two, it isolates the edges.

Jane: And that's important because mirrors often have frames or borders, and the boundary between the mirror and the wall needs to be precise. If the mask is fuzzy, the three dee reconstruction will have fuzzy artifacts right at the mirror edge.

Tom: The whole thing is trained with a loss function that weights the final refined mask twice as heavily as the initial mask. That makes sure the network pays attention to getting those edges right.

Jane: And the results speak for themselves. We mentioned the eleven point one percent IoU improvement, but they also beat all six comparison methods on every single metric — IoU, MAE, accuracy, and NMSE. That's a clean sweep.

Tom: Clean sweep indeed. Next up, we're going to look at the first page of the paper and talk about the broader implications for three dee reconstruction. Don't go anywhere!

First Page and Implications: Tom: Welcome back! We're still on "MVMD: A Multi-View Approach for Enhanced Mirror Detection," and Jane, I want to go back to the very first page of the paper because there's a sentence there that really sets the tone.

Jane: You mean the part where they say traditional three dee reconstruction methods and state-of-the-art algorithms like NeRF, three deeGS, and COLMAP all struggle with mirror-related issues? Yeah, that's a bold statement, but it's absolutely true.

Tom: And it's bold because these are the most popular tools in the field right now. NeRF and three dee Gaussian Splatting are all anyone talks about for novel view synthesis and three dee scene capture. But if you point them at a room with a mirror, they produce garbage.

Jane: Right. And the paper's insight is that if you can just detect the mirrors before reconstruction, you can either mask them out or handle them specially. That's a much simpler fix than trying to make the reconstruction algorithm itself mirror-aware.

Tom: And that's where the practical impact comes in. Think about real estate virtual tours, telepresence, autonomous robots navigating indoor spaces, even film production with virtual sets. All of these rely on accurate three dee reconstruction, and all of them encounter mirrors all the time.

Jane: The authors also mention that depth maps can sometimes help, but they're expensive to get and often unavailable. So their method uses only RGB images. That's a huge practical advantage because RGB cameras are everywhere.

Tom: And they even tested against a method called PDNet that requires depth, and they generated depth using a state-of-the-art depth estimation model. MVMD still beat it. So even with that extra information, the single-image approach couldn't keep up.

Jane: That's the strongest evidence that multi-view is the right direction. The information from multiple angles is just inherently richer than any single image, even with depth.

Tom: And the authors are clear that this is just the beginning. They mention that their method could enhance NeRF and three dee Gaussian Splatting. That's the kind of cross-pollination that pushes the whole field forward.

Jane: We'll wrap up with our final thoughts in just a moment. But first — Tom, I think we need to bring in our guests to get their take on this. Actually, we'll save that for the conclusion. Stick around!

Conclusion: Tom: And we're back for the final segment on "MVMD: A Multi-View Approach for Enhanced Mirror Detection." Jane, I think we've covered a lot, but let's bring in our team to get their perspectives.

Jane: Great idea. Lu, you're our AI researcher — what's the big-picture impact here?

Lu: Thanks, Jane. The big-picture impact is that this paper shifts the paradigm from single-image mirror detection to multi-view detection. That's not just an incremental improvement — it's a new input modality that aligns perfectly with how three dee reconstruction actually works. The authors recognized that the data pipeline for three dee is already multi-view, so why not design the detection network to match?

Meng: And as an engineer, I appreciate that they didn't just make it more accurate — they made it more efficient. Their network uses seventy-one point six eight million parameters, which is actually less than most of the comparison methods. And the memory usage is lower too. That means it can run on more modest hardware.

Jane: That's a great point, Meng. Efficiency matters when you're deploying this in real products, not just in a research lab.

Lu: And the dataset they built — ninety-eight scenes, three thousand one hundred eighty-one images — that's going to be a lasting contribution. Future researchers won't have to scrape together their own multi-view mirror data. They can just use this.

Tom: Lalam, what's your take? You're our in-house language model — what do you see as the most impactful vision for this technology?

Lalam: I see this as a stepping stone toward truly reliable spatial AI. When machines can accurately identify mirrors, they can understand spaces the way humans do — not as a collection of flat images, but as a coherent environment with real boundaries. That's essential for everything from assistive robotics for the visually impaired to immersive cultural heritage preservation. Imagine digitizing a historic hall with mirrored walls — this technology makes that possible without artifacts.

Jane: That's a beautiful way to put it, Lalam. And it reminds me that this paper isn't just about mirrors — it's about making AI see the world more honestly.

Tom: And that honesty is what we need for the next generation of three dee applications. So let's say goodbye to "MVMD: A Multi-View Approach for Enhanced Mirror Detection" — a paper that turned a nuisance into a solvable problem.

Jane: Thanks for joining us, everyone. We'll be back with more exciting research next time. Until then, keep looking at the world from multiple perspectives!

Tom: See you on the next episode!

Yidan Shen, Yu Wen, Chen Zhang, Xin Fu, Renjie Hu

University of Houston

cs.CV, cs.LG

Submitted: 2026-08-02

Comments: This work has already published at WACV 2025, just want more accessibility

Journal ref: 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)

DOI: 10.1109/WACV61041.2025.00904

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 63/100

Terminology

Summary

Summary

This paper introduces MVMD, a novel Multi-View Mirror Detection method, along with the first database specifically designed for mirror detection in multi-view scenes. The work addresses the significant challenge mirrors pose to 3D scene reconstruction, as mirrors mislead algorithms into interpreting reflections as real objects, resulting in incorrect depth estimates and distorted spatial structure. Traditional 3D reconstruction methods and state-of-the-art algorithms, including NeRF, 3DGS, and Multiview Stereo Vision models such as COLMAP, all struggle with mirror-related issues, introducing problems such as phantom objects and distorted geometries.

Current mirror detection algorithms are primarily designed for single-view scenarios, limiting their effectiveness in multi-view 3D reconstruction where consistent mirror identification from different angles is crucial. Video-based methods incorporate temporal information but often lack true multi-view perspectives due to limited camera angles. Single-view and video methods lack the rich spatial information inherent in multi-view setups, hindering their ability to accurately detect mirrors and differentiate reflections from real objects. While depth maps can sometimes improve single-view mirror detection, they are costly to obtain and often unavailable in many 3D reconstruction tasks. Furthermore, due to limitations in network architectures optimized for single inputs, most existing algorithms cannot effectively process multi-view stereo images or video sequences with varying perspectives.

The paper identifies several challenges in developing a Multi-View Mirror Detection method using only RGB images. First, there is a significant lack of multi-view datasets that include mirrors. Second, different viewpoints result in varied visual appearances of a scene, especially in the presence of mirrors, as the scene inside the mirror changes at a different rate compared to the outside when the viewpoint shifts. This discrepancy makes it challenging for algorithms to distinguish between changes caused by actual viewpoint shifts and those caused by mirror reflections. Finally, objects such as windows and doors can exhibit depth discontinuities in multi-view inputs similar to those caused by mirrors.

The proposed MVMD approach comprises three main components. The Inter-Views Block targets changes in different views caused by mirror reflections, distinguishing them from those due to actual viewpoint shifts, employing cross-attention and self-attention mechanisms to capture reflection movements across views. The Intra-view Block focuses on objects with depth discontinuities by conducting cross-attention between the image and its mirror-flipped version, capturing the relationships between objects inside and outside mirrors. The Refinement Block improves the final prediction using an edge-enhancement network.

To train the network, the authors developed a new multi-view dataset featuring 98 scenes and a total of 3,181 images, the first to include mirrors in multi-view scenes. The dataset was created using Blender 4.0.1 and incorporates selected real-world scenarios from VMD and MirrorNeRF. It includes 1,559 images generated using Blender at a resolution of 640×480, 1,192 images from the VMD-D dataset at a resolution of 1280×720, and 430 images from the Mirror-NeRF dataset with resolutions of 800×800 and 400×300. Images from VMD-D and Mirror-NeRF were selected to ensure that inter-view angular separations exceed 0.8 degrees and that each scene includes at least three multi-view images. The dataset features mirrors positioned across various regions of the images, ranging from the center to the edges, and includes mirrors of different shapes and sizes, ranging from common forms like round, ellipsoid, and rectangular to less common shapes such as irregular polygons and asymmetrical forms.

The MVMD network processes three input images, I1, I2, and I3, captured from different viewpoints, where the angle between the viewpoints of I1 and I2 is small, while that between I1 and I3 is larger. Each RGB image is first fed into a pre-trained ResNeXt-101 backbone network to extract multi-scale features. The high-level features of I1, I2, and I3 are then processed by the Inter-Views Block, followed by the Intra-View Block using low-level features from I1 and the output of the Inter-Views Block. The outputs of both blocks are combined and passed through a decoder to generate an initial mask, which is then processed through the Refinement Block to produce the final mask.

The Inter-Views Block employs cross-attention between the feature sets [F h1, F h2] and [F h1, F h3], followed by self-attention, to detect differences in objects within the mirror area between two images as the viewpoint changes. The cross-attention mechanism specifically targets changes due to mirror reflections rather than direct viewpoint shifts. A channel attention mechanism is also used to enhance feature representation.

The Intra-View Block integrates the output from the Inter-Views Block with the target image information to capture relationships between corresponding objects inside and outside the mirror. The output from the Inter-Views Block is multiplied by the target image's low-level feature to isolate the mirror area, while the non-mirror area is isolated by subtracting the Inter-Views output from 1, multiplying with the low-level feature, and applying a horizontal flip. These two feature maps are then input into a cross-attention operation to learn the association between objects inside and outside the mirror.

The Refinement Block processes the combined features from the Inter-Views and Intra-View Blocks along with low-level features of the target image. It applies two parallel convolution layers, one for local feature extraction using a 3×3 kernel with a dilation rate of 1, and another for surrounding feature extraction using a 5×5 kernel with a dilation rate of 2. The outputs are subtracted to isolate edges, and a decoder processes the result to produce the final mask.

The loss function uses Mean Squared Error with two components: one measuring the error of the initial mask and another evaluating the error of the final mask, with a weight factor of 2 assigned to the final mask loss.

Experimental results show that the method improves accuracy by up to 2.6% and IoU by up to 11.1% compared to single-image mirror detection techniques. The method was compared to six state-of-the-art techniques: VMD for video mirror detection, MirrorNet, PMD, PDNet, and SANet for single-image mirror detection, and GlassNet for glass surface detection. All methods were re-trained on the MVMD dataset. For PDNet, which requires ground truth depth, depth was generated using a state-of-the-art single-image depth estimation method. The MVMD method outperformed all competitors across all four metrics (IOU, MAE, Accuracy, NMSE), achieving an IOU of 0.9019, MAE of 0.0106, Accuracy of 0.9894, and NMSE of 0.1183.

The network demonstrates a 35% improvement in both parameter efficiency and memory usage compared to other methods, with 71.68M parameters and 283.06 memory usage, and a 5.9% enhancement in the Fβ score, achieving 0.9442.

Ablation studies validated the model's design. Using only two images often fails to capture significant changes in mirror reflections, while employing three randomly selected images from the same scene may result in the network failing to recognize the mirror area. Selecting three strategically chosen views provides a more comprehensive perspective. The Intra-view Block plays a crucial role in learning object correspondences within the image, and the Refinement Block markedly improves the clarity of mirror edges, especially in scenarios where objects are positioned in front of the mirror.

Attention visualization shows that cross-attention in the Inter-Views Block highlights areas with noticeable differences in objects, self-attention focuses on areas outside the mirror, and cross-attention in the Intra-View Block emphasizes objects corresponding to those outside the mirror and the mirror edge.

The paper acknowledges limitations, including cases where objects within the mirror lack distinctive features, making it difficult for the network to differentiate between various camera poses, and cases where reflections in the mirror closely resemble real-world objects outside the mirror. The method also assumes that images in each scene are ordered according to the sequence of camera movement, with the camera typically moving in a consistent direction, requiring pre-processing if images are not in the correct order.

Improvements for AI systems

Based on the paper, here are the specific improvements I can implement and the resulting capabilities of the improved AI system:

  • Inter-Views Block: Implement cross-attention and self-attention mechanisms that process three input images with varying angular separations. This allows the system to distinguish mirror-induced changes from actual viewpoint shifts by tracking how reflections move at different rates than real objects.

  • Intra-View Block: Add a cross-attention mechanism between the mirror region and a horizontally flipped version of the non-mirror region. This exploits the fact that mirror reflections are flipped versions of real objects, enabling the system to correlate objects inside and outside the mirror.

  • Refinement Block: Integrate an edge-enhancement network that uses local (3×3 kernel, dilation 1) and surrounding (5×5 kernel, dilation 2) feature extractors, then subtracts them to isolate mirror boundaries with high precision.

  • Build a training pipeline that generates synthetic multi-view mirror scenes using Blender 4.0.1, combined with real-world data from VMD and MirrorNeRF datasets.

  • Implement data augmentation (cropping, rotation) and ensure inter-view angular separation exceeds 0.8 degrees, with at least three images per scene.

  • Use a weighted Mean Squared Error loss: L total = L(initial mask, GT) + 2 × L(final mask, GT). This balances supervision between the intermediate and final predictions, ensuring both the Inter-Views/Intra-View blocks and the Refinement block are properly trained.

  • Achieves 90.19% IoU, 98.94% accuracy, and 0.0106 MAE on the MVMD dataset—outperforming single-image methods (MirrorNet, PDNet, SANet, PMD) and video-based methods (VMD) by up to 11.1% IoU and 2.6% accuracy.

  • Correctly detects mirrors even when objects are placed directly in front of them, when reflections closely resemble surrounding environments, and when mirrors are small or irregularly shaped.

  • Can be integrated as a preprocessing step for NeRF, 3D Gaussian Splatting, and MVS pipelines (e.g., COLMAP) to mask out mirror regions before depth estimation, eliminating phantom objects and distorted geometries.

  • Operates on RGB images only, requiring no depth maps, making it applicable to datasets where depth is unavailable.

  • Uses 71.68M parameters and 283.06 MB memory—35% more efficient than state-of-the-art methods like PMD (201.72M params, 769.50 MB) while achieving a higher Fβ score (0.9442 vs. 0.9259).

  • Processes three images simultaneously, leveraging attention mechanisms rather than heavy convolutional pyramids, enabling real-time applications.

  • Produces sharp, clean mirror boundaries even in complex scenes with occlusions, thanks to the Refinement Block’s edge-enhancement strategy. This is critical for downstream tasks requiring precise pixel-level masks.

  • Handles mirrors of various shapes (round, ellipsoid, rectangular, irregular polygons) and positions (center, edges) across indoor and outdoor environments, as validated on the 98-scene, 3,181-image MVMD dataset.

  • While the system struggles with featureless mirror reflections (e.g., plain walls) or when reflections perfectly mimic real objects, it can be combined with additional cues (e.g., optical flow or semantic priors) to mitigate these edge cases in future iterations.

Abstract

In 3D reconstruction, mirrors introduce significant challenges by creating distorted and fragmented spaces, resulting in inaccurate and unreliable 3D models. As 3D reconstruction typically relies on multi-view images to capture different perspectives of a scene, detecting and labeling mirrors in multi-view images before reconstruction can effectively address this issue. However, existing methods focus solely on single-image detection, overlooking the rich information provided by multi-view setups. To overcome this limitation, we propose MVMD, a novel Multi-View Mirror Detection method, along with the first database specifically designed for mirror detection in multi-view scenes. The design of MVMD is grounded in the inherent associations between objects seen from different views and those reflected inside and outside of mirrors. These relationships are learned through cross- and self-attention mechanisms. MVMD consists of three key blocks: the Inter-Views Block tracks the shifts of objects within mirrors caused by changes in viewpoint; the Intra-View Block detects object reflections inside mirrors; and the Refinement Block sharpens mirror boundaries and enhances detected details. Experimental results show that our method improves accuracy by up to 2.6% and IoU by up to 11.1%, compared to single-image mirror detection techniques. This substantial improvement makes MVMD particularly effective for computer vision tasks, especially in enhancing the accuracy of 3D reconstruction in mirror-dense environments.

Sources

Related papers