DIVER: Diving Deeper into Distilled Data via Expressive Semantic Recovery

arXiv:2605.12649 · cs.CV · Submitted 2026-05-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "DIVER: Diving Deeper into Distilled Data via Expressive Semantic Recovery".

Jane: Dataset Distillation (DD) aims to compress massive datasets into compact representations for privacy and efficiency,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, we’re talking about "DIVER: Diving Deeper into Distilled Data via Expressive Semantic Recovery," and the authors are Qianxin Xia, Zhiyong Shu, Wenbo Jiang, Jiawei Du, and Jielei Wang. The title suggests they are going deep into the distilled data using some kind of semantic recovery process.

Jane: That's right. The core idea is that traditional distillation methods can create images that look weird or noisy because they focus too much on the specific architecture used during training, which then hurts how well those models perform when we try to use them elsewhere.

Lu: Exactly! They are proposing a new dual-stage framework where they use a pre-trained diffusion model to recover the actual high-level meaning from that distilled data, filtering out the noise specific to the prior architecture.

Meng: So, instead of just accepting those abstract images, they're using this generative model to refine them into something more meaningful for general AI applications. That’s a big step toward making these compressed datasets truly versatile.

Lalam: If we can get data that has strong intrinsic semantics instead of architecture-specific artifacts, it means the resulting AI systems will be much better at generalizing to new tasks and new model structures.

The paper's summary: Tom: So, summarizing the DIVER paper, they've broken the problem into two parts: Dataset Distillation, which is stage one where they get that initial compact dataset. Then comes Diving into Distilled Data, or DDD, in stage two where the generative model takes those images and refines them into a synthetic dataset.

Jane: That’s a key point. The paper explains that this second stage uses three specific semantic recovery strategies: Semantic Inheritance, Semantic Guidance, and Semantic Fusion to clean up the data.

Lu: Let's talk about those strategies—Semantic Inheritance helps by distilling high-level semantics into latent space to filter out architecture-specific noise and keep the core meaning of the images.

Meng: That sounds like a smart way to regularize things; treating that noise as non-essential information helps guide the generation process toward what's truly important for generalization.

Lalam: And Semantic Guidance improves on that by directing the reverse diffusion process to make sure the label meanings stay close to what they were originally intended, which is super important for maintaining fidelity.

The paper's improvements: Tom: The actual technical improvement centers around decoupling DD into DD and DDD, which means Stage II minimizes the objective across various architectures by synthesizing a new dataset that ensures applicability at all.

Jane: They are trying to ensure the resulting synthetic data works well regardless of whether we use a ResNet or a MobileNet architecture when testing it, which solves that cross-architecture generalization dilemma they mentioned earlier.

Lu: The paper shows how Semantic Inheritance uses the VAE encoder to suppress high-frequency noise, while the diffusion model brings the images back onto the real data manifold, which they say enhances generalization performance.

Meng: I'm interested in the efficiency part; they claim it requires processing time comparable to running a raw DiT on ImageNet at two hundred fifty-six times two hundred fifty-six resolution using only four gigabytes of GPU memory. That’s quite manageable for practical engineering work.

Lalam: And the paper also details how Semantic Fusion is applied only during a specific phase, the Semantic Phase, to fuse those conditional labels with the inherited and guided latents to boost efficiency without introducing artifacts from full-phase guidance.

Conclusion: Tom: So, wrapping up DIVER: this framework successfully proposes a dual-stage method that recovers semantics suppressed by architecture patterns in distilled datasets by using inheritance, guidance, and fusion techniques. It's a plugin to directly optimize the dataset generated by classical distillation without needing access to the original data or training data for the synthesis step.

Jane: Essentially, DIVER aims to enhance cross-architecture generalization by making sure that what we distill actually retains its high-level meaning even when architectures are different. It’s about synthesizing a better dataset from existing distilled images in a raw, training-free way.

Lu: The implication is that we can use these methods to create robust data representations for any AI system without needing to retrain everything from scratch for every new architecture we want to test on.

Meng: For practical deployment, the efficiency metrics they provide suggest this approach is computationally light enough that it doesn't add significant overhead when integrating it into existing training pipelines.

Lalam: I think the biggest cultural impact here is enabling a more reliable pathway for building diverse AI systems, allowing us to focus on creating better applications rather than struggling with data representation bottlenecks.

Qianxin Xia, Zhiyong Shu, Wenbo Jiang, Jiawei Du

University of Electronic Science and Technology of China · Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), Singapore

cs.CV

Submitted: 2026-05-12

Updated: 2026-09-30

Importance score: 79/100

The gist: Dataset Distillation (DD) aims to compress massive datasets into compact representations for privacy and efficiency, but classical DD methods often suffer from learning specific patterns that overfit

Key concepts

Dataset Distillation (DD)
The initial process of compressing massive datasets into compact representations. Classical DD often fails because it learns specific patterns tied to a prior architecture, which hides important high-level meanings needed for good generalization.
Diving into Distilled Data (DDD)
The second stage of DIVER. It takes the initial distilled data and uses a pre-trained generative model to synthesize a new dataset. This process aims to fix the semantic loss from Stage I, creating a synthetic dataset that works better on unseen architectures.
Semantic Inheritance
A strategy where the distilled image is projected into a deep latent code. This inherited code acts as a regularizer, filtering out architecture-specific noise while keeping the essential high-level semantics of the original data intact for better generalization.
Semantic Guidance
A method that steers the reverse sampling process using a guidance function. This ensures that when fusing labels with latents, the resulting semantic information remains close to what was originally present in the distilled dataset.

Terminology

Summary

Dataset Distillation (DD) aims to compress massive datasets into compact representations for privacy and efficiency, but classical DD methods often suffer from learning specific patterns that overfit on a prior architecture, leading to suppressed high-level semantics and poor cross-architecture generalization. This paper proposes DIVER, a novel dual-stage distillation framework that leverages a pre-trained diffusion model to dive deeper into distilled data via expressive semantic recovery. By decoupling the problem into Dataset Distillation (DD) and Diving into Distilled Data (DDD), DIVER synthesizes a new dataset from the distilled images, significantly enhancing generalization capabilities across various architectures.

How it works

DIVER operates in a dual-stage paradigm: Stage I is identical to classical DD, which extracts the initial distilled dataset. The core innovation lies in Stage II, termed Diving into Distilled Data (DDD), where a pre-trained generative model refines the distilled dataset into a synthetic dataset. This process is guided by three semantic recovery strategies designed to recover semantics masked by architecture-specific patterns:

  1. Semantic Inheritance: This strategy distills high-level semantics of abstract distilled images into the latent space to filter out architecture-specific “noise” and retain the intrinsic semantics. It projects the distilled image into a deep latent code and initializes the sampling process with this inherited latent code, serving as a regularization mechanism to constrain the sampling trajectory.

  2. Semantic Guidance: This strategy improves the preservation of original semantics by directing the reverse procedure. It introduces a guidance function, designed according to Equation 8, which aims to maintain the inherited semantics of the distilled dataset so that the fused label semantics stay close to the original semantics.

  3. Semantic Fusion: This is applied only during a specific phase of the reverse process—the Semantic Phase (SP)—to fuse conditional labels with inherited and guided latents. This targeted approach is designed to enhance both sampling efficiency and quality, preventing semantic ambiguity and artifacts that arise from full-phase guidance.

Decoupled Dataset Distillation

The paper formally introduces DDD, which refines the distilled dataset into a synthetic dataset without requiring access to the original dataset. The objective of DDD is defined as:

/S

This decouples classical DD into DD (Dataset Distillation) and DDD (Diving into Distilled Data). While Stage I performs the initial distillation, Stage II minimizes the objective on various architectures by synthesizing a new dataset where:

/S

The goal is to refine the distilled images to ensure their applicability at Φv as well, mitigating the impact of specific architecture patterns like those learned by a prior architecture Φp.

Semantic Recovery Strategies in Detail

The three recovery strategies are integrated into a pre-trained guided diffusion model to unlock the suppressed potential for generalization:

/SI

The VAE encoder (Semantic Inheritance) suppresses high-frequency noise by treating it as non-essential, while the diffusion model recovers semantics by bringing images back to the real data manifold. This joint effect enhances generalization performance.

/SG

Semantic Guidance actively reinforces semantic retention by steering the sampling process, ensuring that the fused label semantics remain close to the original semantics.

/SF

Semantic Fusion integrates conditional labels into inherited and guided latents only during the critical Semantic Phase (SP), which fuses inherited semantics, conditional labels, and guidance, thereby producing clear semantics while preventing artifacts from full-phase fusion.

Experimental Validation

Extensive experiments validate DIVER's effectiveness in improving cross-architecture generalization. The results demonstrate that DIVER consistently improves the performance of the generalization of all methods, with improvements ranging from marginal to substantial across various evaluation metrics and architectures (e.g., ResNet18, ShuffleNet-V2, MobileNet-V2). Specifically:

/Cross-Architecture Generalization

DIVER shows superior performance compared to classical DD methods across different matching strategies (distribution, gradient, trajectory) and generative prior methods like GLaD. For instance, in Table 1 (Tab. 1), DIVER consistently achieves higher performance metrics than classical DD for unseen architectures.

/Efficiency and Scalability

The method is computationally efficient; it requires processing time comparable to raw DiT on ImageNet (256×256) with only 4 GB of GPU memory usage. The computational cost is minimal, as SI incurs negligible overhead, and SG avoids gradient calculations during sampling by setting the guidance term to zero.

Conclusion

DIVER successfully proposes a dual-stage framework that enhances cross-architecture generalization by recovering expressive semantics suppressed by architecture-specific patterns in distilled datasets. It serves as a plugin to directly optimize the distilled dataset generated by classical dataset distillation in a raw data-free and training-free manner, achieving high model performance with minimal storage and computational overhead.

Improvements for AI systems

As a fastidious and diligent researcher, I have analyzed the core contributions of the DIVER framework presented in this paper. Based on its methodology—specifically decoupling classical Dataset Distillation (DD) into Stage I (DD) and Stage II (DDD), leveraging a pre-trained diffusion model for semantic recovery via Semantic Inheritance, Guidance, and Fusion—here are the specific improvements and capabilities this system can bring to AI systems:

The proposed DIVER framework enhances AI systems by creating a robust, architecture-agnostic mechanism for synthesizing high-quality training data from limited original datasets.

Here are the specific improvements and what the improved AI system can achieve:

Sources

Related papers