RefGC-SR squared: Reference-guided Super-Resolution and Refinement of AI Generated Content

arXiv:2606.15158 · cs.CV · Submitted 2026-06-13 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "RefGC-SR squared: Reference-guided Super-Resolution and Refinement of AI Generated Content".

Jane: Reference-guided generation pipelines often discard fine-grained details from high-resolution reference images before generating low-resolution content, leading to artifacts like identity distortion and texture loss.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Moving on, let's talk about who put this together. The authors are Jeahun Sung, Dahyeon Kye, Soo Ye Kim, and Jihyong Oh from CMLab at Chung-Ang University and Adobe Research. They’re the team driving this work on RefGC-SR squared: Reference-guided Super-Resolution and Refinement of AI Generated Content.

Jane: It's interesting to see a collaboration between academic institutions and a major industry player like Adobe Research; that usually means the research is very focused on making something practically usable right away. I wonder what their initial thoughts were about how current reference-guided generation pipelines fall short.

Lu: The authors seem very aware of the limitations in existing methods, specifically how current pipelines discard fine details from the high-resolution reference image before generating low-resolution content, which is where this paper starts its focus.

Meng: I’m thinking about those initial stages; if they are dealing with a problem where fine detail is lost early on, that suggests the architecture needs to be very sensitive to how it handles resolution scaling and information retention right from the start.

Lalam: It shows a dedication to tackling those specific weaknesses in the generation workflow rather than just adding another layer on top of an already flawed process.

The paper's summary: Tom: So, what is the core idea here? Essentially, RefGC-SR squared introduces a new task that combines super-resolution and artifact refinement into one single step. They are taking a low-resolution generated content image and the original high-resolution reference image as inputs to generate an HR image that fixes both problems at the same time.

Jane: That's a really clear explanation; it means instead of doing resolution enhancement and fixing generation errors as separate steps, this model tries to do both simultaneously while making sure the final output looks faithful to the original high-resolution reference object.

Lu: The paper points out that current methods often recover resolution by assuming natural image degradations or they only operate in the low-resolution domain, so RefGC-SR2 is unique because it doesn't rely on those assumptions and instead explicitly addresses both gaps in a single formulation.

Meng: That means they aren't just patching up artifacts after the fact; they are building a mechanism that uses the reference information actively during the upscaling process to guide detail recovery.

Lalam: This moves us toward creating a much more robust system where we can trust the final output to maintain fidelity while achieving high visual quality, which is crucial for any consumer-facing AI application.

The paper's improvements: Tom: Now let’s get into how they actually do this. They introduce two major architectural ideas: first, a frequency-adaptive diffusion transformer model called FreqMoLE, and second, a frequency-based loss function called Lf that guides the training process.

Jane: The FreqMoLE module is interesting because it uses two different LoRA experts—one for low frequencies and one for high frequencies—and they adapt which expert to use based on layer frequency. This suggests a smart way to handle different aspects of the image, like overall structure versus fine textures.

Lu: And then you have the frequency-based loss function, which has two parts: an LLF term that aligns global structure with the HRGT, and an LHF term that transfers fine details directly from the HRRI into the output. That’s a very specific way to balance structural alignment and detail recovery.

Meng: From my side, I see this frequency-based approach as a sophisticated way to ensure we don't just get blurry upscaling; we are explicitly telling the model where it needs to focus its computational power based on what information is most important at different scales.

Lalam: This level of detail in the training objective sounds like it’s how they manage to handle the complexity of simultaneously recovering both global structure and fine textures, which is a tricky balance in these types of tasks.

Conclusion: Tom: So, to wrap things up on "RefGC-SR squared: Reference-guided Super-Resolution and Refinement of AI Generated Content," the main point is that they’ve created a task that handles super-resolution and artifact refinement together by reusing the original high-resolution reference image. This results in an output that preserves identity and recovers fine details, which is much better than previous methods.

Jane: It really boils down to using a dual approach—the FreqMoLE model for selective detail injection and the frequency-based loss function to guide how those details are recovered relative to the reference image. It’s about making sure the upscaling process respects both structure and fine texture simultaneously.

Lu: The implication is that we can finally get a way to produce high-quality, reference-faithful images from generated content without having to run complex, separate refinement passes afterward, which simplifies the overall pipeline significantly.

Meng: Practically speaking, this means faster processing times for users because they don't have to wait for two sequential AI processes; they get a single result that is already high quality.

Lalam: For the broader culture of AI tools, this suggests we can move toward more seamlessly integrated generative workflows where fidelity and resolution are prioritized together right from the start.

Tom: That’s what we have here with RefGC-SR squared; it’s a really smart way to handle that last mile personalization for generated images. We've covered a lot of ground, but we gotta leave you with this idea of how frequency adaptation can be key to unlocking better detail recovery in these models.

Jane: Right, and next time we look at new papers, we’ll see if they adopt similar strategies to combine multiple objectives into one unified task.

Lu: I’m really looking forward to seeing how researchers adapt this specific frequency-aware loss structure in future work to explore even more complex visual tasks.

Meng: For practical implementation, we'll be watching how efficiently they can deploy this model on varied hardware because that dual-expert architecture might introduce some computational overhead we need to manage.

Lalam: I’m optimistic that this direction of combining SR and refinement into one post-processing task sets a strong standard for how we build next generation image tools.

CMLab, Chung-Ang University · Adobe Research

cs.CV

Submitted: 2026-06-13

Updated: 2026-10-06

Project page: https://cmlab-korea.github.io/RefGC-SR2

Importance score: 91/100

The gist: Reference-guided generation pipelines often discard fine-grained details from high-resolution reference images before generating low-resolution content, leading to artifacts like identity distortion

Key concepts

HRRI
High-Resolution Reference Image is the original, detailed image used as a guide. It contains all the fine details of the object that are lost when an image is downsampled for generation. This image is crucial because it provides the ground truth for what the final high-resolution output should look like in terms of texture and detail.
LRGI
Low-Resolution Generated Image is the initial, lower-quality content created by an AI model. This image often contains noticeable flaws or artifacts from the generation process, such as inconsistent textures or distorted features. The goal is to improve this specific input image by refining its quality.
FreqMoLE
Frequency-adaptive Mixture of LoRA Experts is a module that uses two specialized experts, one for low frequencies and one for high frequencies. It intelligently routes the processing task by assigning more weight to the appropriate expert based on the layer of the model, allowing it to handle both global structure and fine details effectively.
LLF and LHF
Low-Frequency Term (LLF) aligns the overall structure of the generated image with a reference image, ensuring correct global shapes. High-Frequency Term (LHF) transfers fine textures from the reference image to the generated image, enhancing detail and correcting artifacts.

Terminology

Summary

Reference-guided generation pipelines often discard fine-grained details from high-resolution reference images before generating low-resolution content, leading to artifacts like identity distortion and texture loss. This paper introduces RefGC-SR2, a new post-processing task that jointly performs super-resolution and artifact refinement by reusing the original high-resolution reference image to recover lost details and refine generative artifacts simultaneously.

The gist

RefGC-SR2 is the first task that jointly satisfies all four criteria: a generated content (GC) input and an HR reference image (HRRI) as inputs, performing super-resolution (SR) and artifact refinement simultaneously.

Problem Addressed

Existing reference-guided generation pipelines downsample the object-centric high-resolution reference image (HRRI) to a fixed low-resolution (LR) before generation, discarding fine details. This loss is compounded by generative artifacts introduced during the generation step, such as identity distortion, detail inconsistency, texture loss, and overall quality degradation. Existing methods either recover resolution assuming natural degradations or operate only in the LR domain. RefGC-SR2 addresses this by taking the LRGI and HRRI as inputs to produce a high-resolution output that preserves the HRRI’s fine-grained information while refining generative artifacts.

Dataset Construction

To train RefGC-SR2, a novel triplet dataset, the RefGC-SR2 Dataset, was constructed. This dataset is defined by three key requirements:

  1. HRRI and HRGT should depict the same object instance in real-world images but differ in view, pose, or scene context.

  2. The input LRGI should be a LR GC image containing generative artifacts as shown in Figure 1.

  3. LRGI should be aligned (with object pose preserved) with the corresponding HRGT, ensuring the model focuses on SR and artifact refinement tasks rather than unintended pose correction.

The construction process involves two stages:

Stage 1: Real-world HRRI-HRGT Pair Curation:

  1. Collecting real-world & Multi-view video data.

  2. Applying Qwen3-VL for filtering object-centric videos (Step 2).

  3. Performing framewise filtering with Qwen3-VL and generating object masks using SAM3 (Step 3).

  4. Human annotators conduct manual verification of the filtering results (Step 4) to obtain high-quality HRRIHRGT pairs.

Stage 2: LRGI Synthesis with DipRefGC:

The DipRefGC (Diptych-Conditioned RefGC Generator) is used to synthesize an aligned LRGI. This generator is built on a frozen FLUX backbone adapted with LoRA-tuned dual ControlNets to impose pose constraints. It uses a diptych formulation with two specialized ControlNets:

  1. Appearance control: The Inpainting ControlNet receives a diptych input where the left panel contains the segmented object from HRRI, and the right panel contains the masked HRGT background and a tight object mask, guiding resynthesis based on HRRI appearance.

  2. Pose control: The Canny ControlNet receives a second diptych input where the right panel contains the HRGT edge map restricted to the object region, enforcing alignment with HRGT.

Model Architecture and Training

The RefGC-SR2 Model is designed with two main components: (i) a frequency-adaptive Mixture of LoRA Experts (FreqMoLE) applied to all DiT blocks, and (ii) a frequency-based loss (Lf).

  1. FreqMoLE: This module replaces a single LoRA expert with two band-specialized experts, a low-frequency (LF) expert and a high-frequency (HF) expert. A routing gate adaptively controls their weights, assigning higher weight to the LF expert in early layers and to the HF expert in later layers, motivated by the layer-wise frequency hierarchy of FLUX-Kontext.

  2. Frequency-based Loss (Lf): This loss is defined as a combination of two terms:

LLF (Low-Frequency Term):

The LF term aligns the global structure with the HRGT: LLF = LF(z0) - LF(zHRGT) M, where · M denotes the l1 norm restricted by an object mask.

This term mainly updates the LF expert through the α-weighted path.

LHF (High-Frequency Term):

The HF term transfers fine detail from the HRRI: LHF = µM HF(z0) - µM HF(zHRRI) + σM HF(z0) - σM HF(zHRRI)

This term mainly updates the HF expert through the (1-α) path.

Improvements for AI systems

Based on the scientific paper RefGC-SR2: Reference-guided Super-Resolution and Refinement of AI Generated Content, here are specific, actionable improvements for existing AI systems and what those improved systems can achieve:

) 1. Implement a Joint Post-Processing Task (RefGC-SR2):

The core improvement is moving beyond sequential processing (first generating, then refining) to a single, unified post-processing task that performs both Super-Resolution (SR) and Generative Artifact Refinement simultaneously.

The improved AI system will take the output of a standard Reference-Guided Generation pipeline (which produces a Low-Resolution Generated Image - LRGI) and the original High-Resolution Reference Image (HRRI) as inputs, then produce a single High-Resolution, artifact-refined output.

This allows for last mile personalization where users can enhance the quality and resolution of an existing generated result while preserving the object's identity faithfully with respect to their reference, without needing to re-run the entire generation process from scratch.

) 2. Reintroduce High-Frequency Details via Frequency-Adaptive Experts (FreqMoLE):

The system should integrate a novel architecture, specifically the FreqMoLE module within Diffusion Transformer (DiT) blocks, which uses a frequency hierarchy to selectively inject fine details.

The improved model will employ two band-specialized LoRA experts (Low-Frequency and High-Frequency) whose routing is controlled by a learnable gating function that emphasizes global structure recovery (LF expert in early layers) and fine detail restoration (HF expert in later layers).

This specifically addresses the texture loss artifact by ensuring that fine details from the HRRI are selectively transferred to the LRGI output, rather than being discarded during standard upscaling.

) 3. Implement Frequency-Based Supervision for Targeted Refinement:

The training objective must be augmented with a frequency-based loss function (Lf) that explicitly guides the model to learn different objectives for different frequency bands.

The system will use a composite loss function, combining Flow-Matching Loss (LFM) for structural alignment with the Frequency-Based Loss (Lf). Lf decomposes latent features into LF and HF components and uses:

  • An LLF term to align the global structure of the LRGI with the HRGT.
  • An LHF term to match channel-wise statistics between the HRRI and LRGI in high-frequency bands.

This allows the model to simultaneously satisfy two goals: learning global structure aligned with the target pose (HRGT) while ensuring fine details are restored using specific information from the reference image (HRRI).

) 4. Construct a Paired Training Dataset for Real-World Artifact Modeling:

The system requires a novel training paradigm that generates ground truth triplets tailored to real-world generative artifacts, rather than relying on synthetic natural image degradations.

The system must utilize the proposed DipRefGC generator to synthesize LRGI inputs that are pose-consistent with the HRGT while explicitly containing realistic generative artifacts (like identity distortion). This creates a dataset of (LRGI, HRRI, HRGT) triplets that accurately reflects the distribution of real user-driven generation pipelines.

This solves the critical problem where naive methods fail because they modify object poses during downsampling or refinement, ensuring the model learns to focus on artifact refinement and resolution recovery rather than unintended pose correction.

This improved AI system can achieve:

  1. Refine generated content (e.g., from text-to-image models) by faithfully restoring fine details (texture, material appearance) lost during low-resolution reference conditioning, while simultaneously increasing the spatial resolution to HR level.

  2. Maintain perfect object identity fidelity by leveraging the high-resolution reference image's specific features, even when the generated content suffers from identity distortion artifacts.

  3. Perform this refinement in a single step (post-processing) rather than as a separate refinement pass, significantly reducing computational cost and improving practical usability for users.

Sources

Related papers