Which Tokens to Merge? Diffusion Dynamics for Efficient Image Generation

summary

The gist

The gist The authors propose Dynamics-aware Token Merging (DaTMe), a training-free framework that grounds token merging in diffusion dynamics by constructing a dynamics-aware importance map from the

In short

The authors propose Dynamics-aware Token Merging (DaTMe), a training-free method to decide which tokens to merge during diffusion model generation. DaTMe uses a map derived from denoising dynamics—combining temporal change and structural features—to identify critical tokens. This approach preserves structurally important regions, like background details, leading to better image quality and coherence compared to prompt-centric methods.

Key concepts

Dynamics-aware Token Merging (DaTMe): A training-free framework that grounds token merging in diffusion model dynamics. It creates an importance map using temporal changes and structural features to adaptively assign tokens into merging roles, ensuring critical regions are preserved while minimizing generation degradation.
Temporal Dynamics Signal: This signal measures how much the predicted clean image structure is still evolving between consecutive denoising steps. It is calculated as the inter-step residual ($|x_0(t) - x_0(t-1)|$), highlighting regions where the model is actively injecting details and should not be merged.
Structural Saliency Signal: This signal captures local high-frequency content, such as boundaries, fine textures, and contours. It is derived from the Tweedie estimate to assign high importance to areas with strong spatial information that anchor visual coherence in the generated image.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 20 episodes
Retrieval-Augmented Generation (RAG) — explained in 19 episodes
Federated Learning — explained in 17 episodes
Flow Matching — explained in 17 episodes
Quantum Fisher Information (QFI) — explained in 17 episodes

This episode discusses

The paper

Which Tokens to Merge? Diffusion Dynamics for Efficient Image Generation · Read on arXiv

SeungJu Cha, Ye-Chan Kim, HyunGee Kim, Sungho Koh, Dong-Jin Kim

Hanyang University

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Which Tokens to Merge? Diffusion Dynamics for Efficient Image Generation".

Jane: The gist The authors propose Dynamics-aware Token Merging (DaTMe),

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, to recap, this paper is about using diffusion dynamics to make smarter token merging decisions for image generation. They are proposing a new way to pick which tokens stay as anchors and which ones can safely be merged without destroying important parts of the image structure.

Jane: Exactly. The authors point out that current approaches based only on prompt relevance fail because they ignore background objects and local details that are not in the prompt, so they introduce this idea that token importance depends on whether a token is still evolving during denoising and if it carries structural information.

Lu: They construct this dynamics-aware importance map from the Tweedie denoised estimate, which combines a temporal signal measuring inter-step changes in the predicted clean image with a structural signal capturing local high-frequency content like boundaries and fine details.

Meng: That sounds complex to implement, but if it works well on models like PixArt-alpha and FLUX, it suggests we could get better quality while still being efficient.

Lalam: They then use this map to adaptively assign tokens into merging roles based on a compression ratio. Important tokens are kept as anchors, while tokens that are both temporally stabilized and structurally homogeneous are selected as safe merge candidates.

The paper's summary: Tom: So, the summary of "Which Tokens to Merge? Diffusion Dynamics for Efficient Image Generation" is that DaTMe is a training-free framework for token merging in diffusion transformers that uses a dynamics-aware importance signal derived from the Tweedie denoised estimate.

Jane: They explain that at each denoising step, DaTMe assigns token roles by looking at the intersection of temporal and structural signals. Tokens with high scores are set as constraint tokens and excluded from merging because they are dynamically or structurally important regions.

Lu: The framework supplements this by taking low-scoring tokens—those from relatively stable and homogeneous regions—and sampling them into grid-based spatial cells to avoid local information loss after merging.

Meng: So, when they hit the budget constraint, like meeting a target compression ratio r, DaTMe picks the right tokens to keep and which ones can be safely merged in those stable areas.

Lalam: After assigning the source set with that additional grid sampling, they use locality-constrained matching to merge those source tokens only within nearby regions. This prevents merging distant tokens from collapsing spatially and distorting local structure.

The paper's improvements: Tom: The main improvements they highlight are that DaTMe better preserves background integrity compared to the CFG-based merging methods that people have used before.

Jane: That’s a big deal because the authors say that CFG's focus on text relevance leads to a blind spot where low-ranked tokens in non-prompt regions get merged into neighboring homogeneous areas, losing fine details.

Lu: They also point out an improvement in their temporal importance signal. Instead of using raw score slope, which they say is heavily dominated by noise schedule scaling, they define the temporal map as the inter-step residual in predicted clean-image space, defined as M(t)temp = xˆ0(t) − xˆ0(t − one) <ref:2610.22370#pg1>.

Meng: That sounds more robust for real inference because it measures how much the predicted structure is still evolving spatially, rather than just the raw score. It gives them a better signal for when to keep or merge tokens.

Lalam: They also show that jointly modeling temporal evolution and structural saliency gives them this dynamics-aware token importance signal that guides merging decisions at the intersection of those two signals.

Conclusion: Tom: To wrap up, the authors argue that DaTMe successfully grounds token merging in the underlying generation dynamics by using a spatiotemporal importance map to identify safely mergeable tokens while preserving structurally critical regions.

Jane: This approach shows that diffusion dynamics are an effective way to make token merging decisions for efficient image generation, and they demonstrate strong visual fidelity across the entire image on models like PixArt-alpha and FLUX.

Lu: The final contribution is that by using this spatiotemporal importance map, DaTMe achieves a strong preservation of image-wide details and state-of-the-art performance compared to prior methods.

Meng: From an engineering standpoint, the fact that they maintain visual fidelity across the whole image while being more efficient is exactly what we need for production models.

Lalam: And I'd add that their experiments confirm that jointly using temporal and spatial signals yields the best overall performance, while replacing just the temporal signal with raw score slope consistently degrades all evaluation metrics.

Tom: So, in summary, this paper "Which Tokens to Merge? Diffusion Dynamics for Efficient Image Generation" successfully grounds token merging in generation dynamics by leveraging a spatiotemporal importance map to identify safely mergeable tokens while preserving structurally critical regions.

More episodes

← Home