Lens Flare Removal and Reconstruction

arXiv:2609.39527 · cs.CV, cs.GR · Submitted 2026-09-30 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Lens Flare Removal and Reconstruction".

Jane: Lens flares are artifacts caused by unintended light paths passing through lenses when observing bright light sources,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Well, folks, we're diving into the paper "Lens Flare Removal and Reconstruction." This research tackles a really tricky problem: getting rid of those big, distracting lens flares in images while still keeping the actual scene information intact for three dee reconstruction.

Jane: That's right, Tom. The title itself tells us it's not just about cleaning up an image; it’s about reconstructing the whole scene by dealing with those artifacts intelligently. It suggests a way to separate what’s real from what the lens is doing, which is pretty fascinating for any visual AI system we build.

Lu: I find the concept of decomposing scenes into separate flare components and scene Gaussians really interesting because it gives us a structured way to handle complex lighting effects that used to be totally intractable. We're moving beyond just removing small flares around a light source to handling those massive, full-frame ones that existing methods just couldn't manage.

Meng: From an engineering standpoint, the idea of explicitly modeling the flare separately from the scene geometry sounds like it could lead to much more robust reconstruction pipelines. We need a method that doesn't just mask things out but understands *why* something looks that way.

Lalam: I think this work points toward an AI system that can handle complex visual inputs with much higher fidelity, which really improves the quality of digital content we create. If we can reliably separate the flare from the scene, it opens up new avenues for image editing and transfer applications in a way that's hard to achieve now.

Tom: Exactly, Lalam. And what they propose is a unified approach using a diffusion model fine-tuned with LoRA to handle both scattering and reflective flares, which is a big step up from just focusing on synthetic data. We're talking about training this on real-world large reflective flares mixed with procedural ones to make the removal model much more versatile.

Jane: That training methodology sounds very practical, Tom. Using an additive linear-radiance composition model like I = Iscene + Iflare / γ for corruption seems like a solid way to introduce flares into the training data. It gives the AI a clear mathematical framework to learn from during fine-tuning.

Lu: The way they're using LoRA on a diffusion model to achieve this kind of flare removal is clever because it allows for efficient adaptation without needing to retrain the entire massive base model, which is a huge computational advantage. It’s an elegant way to inject specialized knowledge about flare artifacts into a general-purpose generative tool.

Title and authors: Meng: So, if the removal model is this good at separating scattering and reflective flares, how does that translate to the reconstruction part? I'm curious about the practical performance on actual three dee scenes.

Tom: That’s where things get really exciting, Meng. They pair this removal model with a dynamic Gaussian splatting model to represent those flares in three dee. They aren't just deleting the flare; they are creating a consistent three dee representation of it, anchoring these Gaussians near the camera's principal point instead of letting them float randomly in world space.

Jane: So, instead of just removing the visual noise, they are giving those flares a defined location and shape in three dee space so the system can reconstruct them consistently. That's a very smart way to handle the view consistency issue that plagues many three dee methods.

Lalam: This three dee representation is key for future applications, Jane. If we have a consistent three dee flare model, we can actually transfer those flares onto entirely new images or even new three dee scenes later on. It’s like having a reusable asset for lighting effects.

Lu: And the way they constrain these Gaussians based on the symmetry of lens flares about the camera's principal point, using a hash-grid MLP to modulate them, shows a deep understanding of how these optical phenomena behave. They are imposing physical constraints derived from optics onto the generative process.

Tom: Speaking of constraints, they also used an off-the-shelf detection pipeline like GroundedSAM2 to find the initial 2D light source locations, which they then refined using TRASE fourteen to get consistent three dee centroids. It shows they're tackling the detection and aggregation problem systematically before feeding it into the main pipeline.

Jane: And that refinement step is important because those initial 2D detections can be quite noisy, so aggregating them into a consistent set of three dee light sources via TRASE ensures the input for their next stage is much cleaner. It shows they aren't just blindly trusting the detection output.

Meng: That systematic approach to light source detection and correction seems very grounded in reality, Jane. How do you handle cases where multiple bounding boxes overlap on the same source before applying TRASE?

Jane: They handle that by keeping the box with the smallest area when multiple detections overlap on a single source, which helps select the most precise location estimate. This small detail shows they're paying attention to robustness in their preprocessing steps.

Title and authors: Lu: Now, for the main optimization pipeline, they jointly optimize the flare representation model and a three dee Gaussian splatting model by training the three deeGS on images generated by their flare-removal model. This end-to-end joint optimization is where they tie everything together, ensuring the scene reconstruction is inherently linked to the flare removal process.

Tom: And that loss function they use is pretty comprehensive, minimizing the difference between the reconstructed scene and ground truth flare-free image alongside a perceptual loss. It’s a multi-faceted objective that aims for both accuracy in pixels and visual similarity in structure.

Lalam: This joint optimization means they are learning to represent the scene and the flare simultaneously, which is powerful because it forces the three deeGS model to learn what the *scene* is while simultaneously learning where those specific *flare* components live. That synergy could lead to much more coherent three dee outputs overall.

Jane: It sounds like a very tightly coupled system, Lu. This coupling allows them to achieve a decomposition of the three dee scene into these two distinct components, which is what makes this approach so unique.

Lu: Indeed, and the final result is that they can reconstruct the flare explicitly and re-render it, which means that if you want to edit or transfer a flare later, you have a discrete three dee object to manipulate. They also showed it works well for real-time rendering, suggesting practical viability.

Tom: And the robustness testing is something I really liked—they incorporated two correction mechanisms for light location errors, a 2D deformation based on camera location and finer deformations that allow for asymmetries. They showed that with at least one of these corrections, the model degrades gracefully, but removing both causes a significant drop in quality.

Meng: That robustness finding is critical for me. If the system can handle localization errors gracefully, it means it's much more deployable in real-world scenarios where light source detection isn't perfect. It moves the technology closer to being usable outside of perfectly controlled lab settings.

Jane: That graceful degradation is a very reassuring result for any practical AI system, Meng. It means we don't have to build impossibly perfect detection systems just for the reconstruction to work.

Title and authors: Lalam: From my perspective, this level of robustness suggests that we can eventually build tools where users can freely edit captured flare patterns and apply them seamlessly to new content without worrying about the underlying three dee structure breaking. That capability for transfer is what really excites me.

Tom: So, to wrap up on the improvements, we have a system that can handle complex lighting artifacts and provides a way to store flare patterns consistently in three dee space. This allows for explicit editing and transfer of those flares onto novel scenes.

Jane: It seems the core improvement is this decomposition capability, allowing us to control the scene elements independently from the lens artifacts. This separation is what makes this reconstruction pipeline distinct.

Lu: The way they achieve three dee-consistent transfer of a captured flare to new scenes, where appearance is queried at the source viewpoint by design, shows a real innovation in how we model optical phenomena. It’s moving beyond just static 2D removal.

Meng: I see the practical implication here being the ability to use these reconstructed flare models for rendering new assets or compositing them into novel environments efficiently, which is a huge time saver in creative pipelines. It’s about efficiency in content creation.

Lalam: And for culture, this means we could see AI-generated visual effects become much more malleable and controllable, allowing for artistic expression that incorporates realistic lighting effects in a new way. It’s about expanding the creative space.

Tom: So, to summarize the whole paper "Lens Flare Removal and Reconstruction," we have a model that uses a diffusion-based approach with LoRA fine-tuning to remove complex flares, and it couples this with a dynamic Gaussian splatting model to create three dee flare representations. This enables high-fidelity, decomposed three dee scene reconstruction that lets us explicitly edit and transfer those captured flare patterns onto new images or scenes.

Jane: That’s a solid summary, Tom. The key is that they are not just removing the noise; they are giving the noise a structured, manipulatable three dee form that is tied to the original viewing conditions. This structure is what makes it useful for downstream tasks like editing and transferring.

Lu: I think the future work will be in pushing this even further into real-time applications, given the success they’ve shown with real-time rendering capabilities mentioned in Table one. We could see these decomposition techniques applied to faster inference engines.

Meng: From a practical standpoint, I'm looking forward to seeing how easily this pipeline integrates into existing production workflows without requiring massive computational overhead for the reconstruction step. We need it to be fast enough for actual use, not just a research paper.

Title and authors: Lalam: I think this technology has the potential to help democratize high-quality visual effects because it makes complex lighting effects accessible and controllable through AI pipelines. It opens up a lot of possibilities for creative applications in film and gaming.

Tom: Fantastic points, everyone. So, to wrap up on the paper "Lens Flare Removal and Reconstruction," we have a model that uses a diffusion-based approach with LoRA fine-tuning to remove complex flares, and it couples this with a dynamic Gaussian splatting model to create three dee flare representations. This enables high-fidelity, decomposed three dee scene reconstruction that lets us explicitly edit and transfer those captured flare patterns onto new images or scenes.

Lalam: I think this technology has the potential to help democratize high-quality visual effects because it makes complex lighting effects accessible and controllable through AI pipelines. It opens up a lot of possibilities for creative applications in film and gaming.

Tom: That’s a solid summary, Jane. The key is that they are not just removing the noise; they are giving the noise a structured, manipulatable three dee form that is tied to the original viewing conditions. This structure is what makes it useful for downstream tasks like editing and transferring.

Jane: It’s a solid summary, Tom. The key is that they are not just removing the noise; they are giving the noise a structured, manipulatable three dee form that is tied to the original viewing conditions. This structure is what makes it useful for downstream tasks like editing and transferring.

The paper's summary: Tom: So, to wrap up on "Lens Flare Removal and Reconstruction," they've built a system that doesn't just get rid of lens flares in an image; it actually breaks down the scene into its core geometry and those distracting light artifacts, giving us a way to handle them separately.

Jane: That’s right, Tom. The paper proposes using a diffusion model for removal and then pairing that with a dynamic three dee Gaussian Splatting model to represent the flares in space, which is really smart because it gives us something tangible to work with.

Lu: I think the core idea is that they’re not just cleaning up pixels; they are creating an explicit three dee representation of what the lens is doing, which opens up incredible avenues for manipulation. This moves us from just image processing to true scene understanding.

Meng: From an engineering standpoint, that explicit decomposition means we get a clean separation between the geometry and the light effects, which should make downstream tasks much more reliable for applications like three dee modeling. I wonder how computationally demanding that joint optimization process really is in practice.

Lalam: From my perspective as a model, this ability to represent flares as manipulatable three dee Gaussians means we can create reusable flare assets that can be edited and transferred onto completely new images or three dee scenes with high fidelity. That’s a huge cultural impact because it makes realistic visual effects much more accessible for everyone to build upon.

Tom: Exactly, Lalam. It’s not just removing the noise; it's giving that noise a structured, manipulatable three dee form tied to the original viewing conditions, which is what makes this reconstruction pipeline so useful for editing and transferring.

Jane: And the paper’s success lies in how they handle those errors during the process; they showed that even if there are initial mistakes in locating a light source, the system corrects them gracefully with built-in mechanisms. That robustness is key for making this technology viable outside of a perfect laboratory setting.

Lu: The way they constrained those three dee flare Gaussians based on the symmetry of lens flares about the camera's principal point, using that hash-grid MLP to modulate them, shows a deep understanding of how these optical phenomena behave. It’s like they’ve essentially programmed the AI with some basic physics constraints.

Meng: I do see the potential for real-time rendering mentioned in Table one but I'm still focused on how this whole decomposition fits into existing production workflows without requiring a massive computational overhead for that reconstruction step. Can we get an idea of the practical speed when integrating this into a live feed?

Lalam: The potential here is huge for creative applications; this means we could see AI-generated visual effects become much more malleable and controllable, allowing for artistic expression that incorporates realistic lighting effects in a new way. It’s about expanding the creative space beyond what's currently possible.

Jane: That malleability is exactly what we need to talk about next; if we can control the flare structure so precisely, it fundamentally changes how artists approach lighting and compositing in digital media.

Tom: Right, so we’ve got a model that cleanly separates the scene from the flares, a three dee representation for those flares that's view-consistent, and tools to edit and transfer those flare patterns onto new content. That’s the main takeaway here.

The paper's improvements: Tom: So, we’ve talked about how they tackle those big lens flares by separating them from the scene using a diffusion model and then giving them a consistent three dee representation for manipulation; now let's talk about what they suggest to make it even better.

Jane: What they are suggesting is actually quite interesting because they’re focusing on making that flare representation dynamic, meaning its shape and position can change based on the camera or the light source location during a transfer. That’s a big step for consistency across different views.

Lu: I find that dynamic aspect fascinating because it means we aren't just storing a static flare; we're modeling how that flare looks from every possible angle relative to its source, which is much more physically accurate for rendering. It pushes the frontier of scene representation beyond simple geometry.

Meng: From an engineering standpoint, making the representation dynamic sounds powerful because it allows for real-time updates or view-dependent modifications without needing a whole new three dee reconstruction every time something moves in the scene. I wonder how they manage that computational overhead during a live rendering scenario.

Lalam: The implication here is that we move beyond just storing an image artifact; we’re creating a reusable, intelligent visual element that can be placed into any new scene and adapt its appearance to match the viewing conditions there. This opens up incredible possibilities for creative pipelines where lighting effects are part of the dynamic environment.

Tom: That adaptability is exactly what makes it so exciting, Lalam. It suggests we could eventually have AI-driven tools where users can dial in the exact flare effect they want for a specific scene configuration, not just pick one pre-baked option.

Jane: And I think that control is what really matters; if we have precise control over how the flare behaves based on its relationship to the camera and light, it makes the whole reconstruction pipeline feel much more intentional rather than just an automated cleanup process.

Lu: The authors also hint at future work focusing on pushing this toward real-time applications, which means they’re already thinking about how this structured decomposition can be sped up for faster inference engines. That shows a good vision for scaling the technology beyond static image processing.

Meng: I agree with Lu; the mention of real-time rendering is crucial because if we can't run this fast enough, it won't integrate into production workflows at all, regardless of how good the underlying math is. We need to see that speed realized in their next steps.

Lalam: For culture, this kind of precision in visual effects could democratize high-quality lighting design; it means artists who aren't steeped in complex rendering engines can now control these sophisticated light interactions just by manipulating the flare primitives. That’s a massive shift for creative expression.

Tom: So, to wrap up on the improvements, we have a system that cleanly separates the scene from the flares, a dynamic three dee representation for those flares tied to viewing conditions, and tools to edit and transfer those flare patterns onto new content with high fidelity. That’s what they are aiming for next.

Conclusion: Tom: So, to wrap up on "Lens Flare Removal and Reconstruction," we’ve seen how they use diffusion models for removal and dynamic three dee representations for flares to get a high-fidelity, decomposed scene reconstruction that lets us explicitly edit those flare patterns onto new images.

Jane: That’s right, Tom. The core idea is giving those visual artifacts a structured, manipulatable form in three dee space that's tied directly to the original viewing conditions of the capture. This structure is what makes it useful for downstream tasks like editing and transferring, which is pretty powerful stuff.

Lu: I think the main implication here lies in how they’ve managed to achieve this view-consistent representation, which is a major hurdle in three dee reconstruction that most methods haven't solved yet. It opens up new ways to model complex optical phenomena like lens flares in digital scenes.

Meng: From an engineering standpoint, the fact that they showed graceful degradation when they had to correct light source localization errors means this pipeline is much more deployable for real-world applications than we might have expected, provided the error isn't catastrophic. We just need to keep pushing those speed metrics.

Lalam: For culture, this technology means we could see AI-generated visual effects become much more malleable and controllable; it allows for artistic expression that incorporates realistic lighting effects in a way that’s highly intentional and repeatable. That’s a huge expansion of what we can create with generative tools.

Tom: Exactly, Lalam. It really shifts the focus from just generating pretty pictures to actually controlling the physics behind those pictures, which is where the real creative power is now going to be.

Jane: So, in simple terms, they’ve built a system that separates what's real scene geometry from what's lens noise, and they’ve given that noise a consistent three dee shape so we can work with it. It’s a big step for image processing.

Lu: The future work on pushing this toward real-time applications is where the wild possibilities really lie; if we can make this fast, we could see these decomposition techniques applied to much more complex, dynamic scenes in live environments.

Meng: I hope they can deliver that speed because right now, the reconstruction step itself seems quite intensive for practical deployment. That’s a big hurdle for adoption in the industry.

Lalam: And if we succeed with this kind of control over visual elements, it could help democratize high-quality visual effects by making complex lighting interactions accessible to a wider range of creators.

Tom: Alright folks, that wraps up our deep dive into the paper "Lens Flare Removal and Reconstruction." It’s been an incredible look at how AI is tackling complex optical problems.

Jane: It really has been fascinating to see how they managed to handle both the removal and the reconstruction aspects of lens flares so neatly.

Lu: I’m still buzzing about the potential for dynamic representations in three dee scene modeling; we could see applications in everything from virtual production to immersive environments.

Meng: I'll be watching their next steps closely regarding computational efficiency; that’s where the real engineering challenge lies moving forward.

Lalam: I think this work really shows how AI can become a powerful tool for artists, expanding the creative language we have in digital media.

TARUN YENAMANDRA, JONATHON LUITEN, DANIEL CREMERS, NATHAN MATSUDA

Meta Reality Labs Research

cs.CV, cs.GR

Submitted: 2026-09-30

Updated: 2026-09-30

Comments: 20 pages, 14 figures. Project page: https://lensflare-3dgs.pages.dev

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 79/100

The gist: Lens flares are artifacts caused by unintended light paths passing through lenses when observing bright light sources, and this work introduces a novel pipeline that decomposes 3D scenes into lens

Key concepts

Lens Flare Removal Model: A unified model fine-tuned via LoRA on a diffusion model that handles both scattering and reflective flares. It is trained by corrupting real images with flare data using an additive linear-radiance composition model to learn how to subtract the flare component.
Dynamic Gaussian Splatting Model: A 3D representation where lens flares are modeled as Gaussians anchored near a near-plane, constrained by symmetry around the camera's principal point. A learned deformation field modulates these canonical Gaussians based on camera and light locations for accurate reconstruction.
Decomposed Reconstruction Pipeline: A joint optimization process that trains a 3D Gaussian Splatting model alongside the flare removal model. The scene is broken down into flares and scene Gaussians, allowing the system to reconstruct both components simultaneously from images.
Light Source Detection and Correction: Uses an open-vocabulary detector (GroundedSAM2) to find 2D light sources, filters them by brightness, and aggregates them into consistent 3D centroids using TRASE. It includes correction mechanisms ($\delta l$, $\delta f$) to handle localization errors gracefully.

Terminology

Benchmark — 37× in this paper · explained in 2 episodes
Ablation — 25× in this paper · explained in 1 episode
Gaussian Splatting — 17× in this paper · explained in 4 episodes
Diffusion Model — 15× in this paper · explained in 8 episodes
Robustness — 12× in this paper · explained in 5 episodes
Fidelity — 11× in this paper · explained in 2 episodes
Downstream Tasks — 8× in this paper · explained in 2 episodes
Fine-Tuning — 8× in this paper · explained in 5 episodes
Core — 5× in this paper · explained in 2 episodes
Loss Function — 5× in this paper · explained in 2 episodes
3D Gaussian Splatting — 4× in this paper · explained in 2 episodes
Computational overhead — 4× in this paper · explained in 2 episodes
Adaptation — 2× in this paper · explained in 1 episode
Aggregation — 2× in this paper · explained in 1 episode
Consistency — 2× in this paper · explained in 3 episodes
Diffusion Models — 2× in this paper · explained in 6 episodes
Graceful Degradation — 2× in this paper · explained in 2 episodes
Latent Diffusion Model — 2× in this paper · explained in 2 episodes
LoRA Adapters — 2× in this paper · explained in 3 episodes
LoRA Fine-Tuning — 2× in this paper · explained in 3 episodes
Accuracy — 1× in this paper · explained in 1 episode
Adam — 1× in this paper · explained in 2 episodes
Adaptability — 1× in this paper · explained in 1 episode
Additivity — 1× in this paper · explained in 1 episode
CVF International Conference on Computer Vision — 1× in this paper
Clustering — 1× in this paper · explained in 2 episodes
Conference on Computer Vision and Pattern Recognition — 1× in this paper
Grounding — 1× in this paper · explained in 3 episodes
Physical Constraints — 1× in this paper · explained in 7 episodes
Provenance — 1× in this paper · explained in 3 episodes
Synthetic Data Generation — 1× in this paper · explained in 5 episodes
Transformer — 1× in this paper · explained in 2 episodes
Variational Autoencoder (VAE) — 1× in this paper · explained in 3 episodes
World Model — 1× in this paper · explained in 7 episodes
alpha — 1× in this paper · explained in 1 episode

Summary

Lens flares are artifacts caused by unintended light paths passing through lenses when observing bright light sources, and this work introduces a novel pipeline that decomposes 3D scenes into lens flares and scene Gaussians to enable consistent flare removal and reconstruction. This approach is significant because it addresses the limitations of existing methods that struggle with large flares and provides a view-consistent 3D representation of flares, which is valuable for applications like editing and transferring captured flare patterns onto novel images or new 3D scenes.

Lens Flare Removal Model

The authors propose a unified lens flare removal model fine-tuned via LoRA on a diffusion-based model (Difix3D+) to handle both scattering and reflective flares. This model is trained on a novel dataset combining publicly available real-world data with procedural generation, specifically focusing on large reflective flares. The training involves corrupting images by adding flare images from the lens flare datasets (both synthetic and real) to an image dataset like Flickr24K, using an additive linear-radiance composition model: I = Iscene + Iflare / γ. The model is fine-tuned using a loss function that minimizes the difference between the output image and a given flare-free image: Ldiff(thetal e, thetal, thetal d) = Ig − Io 22 + LPIPS(Ig, Io).

Lens Flare Representation and Reconstruction

To represent lens flares in 3D, the authors adopt a dynamic Gaussian splatting model. Lens flares are modeled as Gaussians anchored to a near-plane in front of the scene rather than free-floating geometry in world space. The representation model is designed per light source based on the symmetry of lens flares about the camera’s principal point, constraining all Gaussians to lie on the projected 2D line connecting each light source to that principal point. A deformation field is learned to modulate these canonical Gaussians based on camera and light locations, using a hash-grid MLP.

Light Source Detection and Correction

The model relies on an off-the-shelf open-vocabulary detection pipeline (GroundedSAM2) to obtain initial 2D light source locations. This detection front end is refined by discarding detections whose mean brightness falls below 190 out of 255 (to remove nonemissive objects) and by keeping the box with the smallest area when multiple bounding boxes overlap on the same source. These noisy 2D detections are then aggregated into a consistent set of 3D light sources per scene using TRASE [14] to obtain their centroids, which are projected onto images to obtain the 2D light positions for training.

Decomposed Reconstruction Pipeline

The authors propose a pipeline that jointly optimizes the flare representation model and a Gaussian splatting (3DGS) model. The 3D scene is decomposed into lens flares and the scene itself by training the 3DGS model on images generated by their flare-removal model. The loss function for this joint optimization includes terms for both reconstruction fidelity of the scene-only image (I s) and the composite image (I), minimizing L = λI s − I o 1 + (1 − λ) / 1 − SSIM(I s, I o) + λI − Iinpt 1 + (1 − λ) / 1 − SSIM(I, Iinpt).

Applications and Robustness

The reconstructed flare is explicit and re-renderable, allowing it to be edited and transferred to novel images or new 3D scenes. The pipeline demonstrates robustness to errors in automatic light-source localization by incorporating two correction mechanisms: a 2D deformation of the light location based on camera location (δl) and finer deformations (δf) that allow for asymmetries. Testing shows that with at least one correction, the model degrades gracefully; however, removing both corrections causes a significant drop in both composite and flare PSNR. The representation also allows for transfer to novel scenes by conditioning the flare MLP on the source camera and light positions during transfer time.

Conclusion

The work introduces a lens-flare removal model and the first lens-flare representation reconstructed from captured multi-view images, demonstrating state-of-the-art performance on established benchmarks and showing that their models consistently remove flares without residual flares while enabling high-quality, decomposed 3D scene reconstruction. The resulting models are designed to be useful for downstream tasks such as flare transfer and editing.

The gist: A novel pipeline decomposes 3D scenes into lens flares and scene Gaussians using a diffusion-based removal model and a view-consistent 3D flare representation, enabling consistent flare removal, high-fidelity decomposed reconstruction, and the transfer of captured flares to novel images or scenes.

Improvements for AI systems

As a fastidious and diligent researcher, I have analyzed this paper, Lens Flare Removal and Reconstruction, which introduces a novel pipeline combining diffusion-based flare removal with a 3D Gaussian Splatting (3DGS) representation for scene decomposition.

Here are the specific improvements to AI systems that can be derived from this research:

A robust, view-consistent 3D Scene Reconstruction System capable of handling complex lighting artifacts.
An Image Enhancement Pipeline capable of removing large, complex lens flares without compromising underlying scene geometry or light source information.
A Novel Flare Representation Module for 3D Scenes that allows for the consistent storage and manipulation (editing/transfer) of captured lens flare patterns across different camera viewpoints and novel target scenes.

The specific capabilities of these improved AI systems are as follows:

A robust, view-consistent 3D Scene Reconstruction System: This system can take a set of multi-view images containing significant lens flares and output a 3D representation (using Gaussian Splatting) that accurately models the underlying scene geometry while explicitly separating it from the non-3D-consistent flare artifacts. This is superior to existing methods that either bake flares into the scene or ignore them entirely.
An Image Enhancement Pipeline: This pipeline can take an input image corrupted by complex, large lens flares (both scattering and reflective) and output a high-quality flare-free image. Crucially, this removal model is trained on real-world datasets of large reflective flares and demonstrates superior performance over existing methods in handling both scattering and reflective types.
A Novel Flare Representation Module: This module allows for the creation of flare primitives (3D Gaussians) anchored to a scene's near plane, conditioned on the camera and light positions. This enables:
Editing and Transfer of Lens Flares: The reconstructed flare Gaussians can be queried at any desired viewpoint and transferred onto novel 2D images or entirely new 3D scenes (e.g., VFX compositions) while preserving their learned shape, color, and orientation, effectively allowing artists to manipulate captured lens flares with high fidelity.
Decomposed Reconstruction: The system provides a clear separation between the Scene Gaussians and the Flare Gaussians, allowing for precise control over which component is used for downstream tasks (e.g., using only scene Gaussians for pure geometry reconstruction, or using both components to preserve flare effects during rendering).

Abstract

The presence of lens flares in images can significantly reduce the quality of downstream application results for tasks such as 3D scene reconstruction. This is because lens flares are a property of the camera imaging system, and not a part of the underlying scene being modeled. There are previous methods that tackle the removal of small flares focused around a light source. However, existing methods struggle with large flares, such as those that fill the entire image. In this work, we compile a novel dataset for large-flare removal, combining publicly available real-world data with a procedural generation pipeline. We fine-tune a diffusion-based model on our dataset to remove complex, large lens flares. On the other hand, lens flares remain effective artistic tools, widely used in the media. While there are ways to simulate 2D flares, representing and reconstructing lens flares consistently across multiple views has not yet been explored. To achieve this, we introduce a flare representation model that leverages the symmetry of lens flares about the camera's principal point. We propose a computational pipeline to jointly optimize this flare model and a Gaussian splatting model (3DGS). This enables the decomposition of a 3D scene into lens flares and the scene itself, using our flare-removal model. Because the reconstructed flare is explicit and re-renderable, it can be edited and transferred to novel images and new 3D scenes. We evaluate removal on an established benchmark and a new one for large reflective flares, quantify the flare/scene decomposition directly, and show that the pipeline is robust to errors in automatic light-source localization.