Fine-Grained Caching for Diffusion Transformers with Few Calibration Conditions

arXiv:2512.05134 · cs.CV, cs.DC, cs.LG · Submitted 2025-11-29 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Fine-Grained Caching for Diffusion Transformers with Few Calibration Conditions".

Tom: InvarDiff presents a training-free, cross-scale caching scheme for DiT-family diffusion generators by exploiting relative feature invariance across timestepscale and layer-scale.

Jane: First, who's behind it and why it matters.

Paper summary: Tom: Welcome back everyone! We're diving into some really interesting work today with "Fine-Grained Caching for Diffusion Transformers with Few Calibration Conditions." This paper tackles the slow nature of diffusion models by introducing a training-free caching scheme that exploits feature invariance. It claims this method can speed up these complex models significantly without needing any retraining on the specific model itself.

Jane: It sounds like they are trying to figure out how to make these iterative processes much faster by intelligently reusing parts of the network based on how features change across different timesteps and layers. The core idea seems to be finding deterministic patterns in those changes so the AI can decide exactly what to keep and what needs recomputing.

Lu: What really catches my eye is how they derive this binary cache plan matrix using feature change statistics derived from a few deterministic runs instead of needing massive amounts of training data <ref:2512.05134#pg0>. That moves the complexity away from iterative fine-tuning and into a more principled planning process.

Meng: From an engineering standpoint, I'm curious about how robust this plan is when we move from the calibration set to real-world deployment on different model architectures or different data distributions. How do we ensure that those quantile thresholds they use hold up in production environments?

Lalam: If this technique can dramatically reduce the sequential evaluation time of these diffusion generators, it opens up possibilities for much more interactive and responsive applications in the future <ref:2512.05134#pg0>. Imagine real-time synthesis capabilities that weren't possible before.

Tom: Exactly, Lalam! It’s about moving from slow sequential evaluation to something that leverages structural invariance to skip computation entirely on many steps <ref:2512.05134#pg0>. Jane, can you explain what those feature change statistics are that drive this caching decision?

Jane: Well, the paper uses two specific measures for temporal invariance across adjacent inference timesteps within the DiT modules, specifically looking at the MHSA and FFN components <ref:2512.05134#pg1>. They look at both the magnitude or energy changes using MSEZ, and they also examine directional changes with cosine similarity to capture how the token representations are shifting between steps <ref:2512.05134#pg1>.

Lu: Those metrics—MSEZ and cosine similarity—are clever because they look at both the size of the change and the direction of that change, which should give a more complete picture of what's happening in the model's internal state <ref:2512.05134#pg1>. It’s not just about one aspect; it captures a broader sense of feature stability across time.

Meng: But I still worry about the practical constraints, Tom. The paper mentions that every evaluation currently invokes expensive modules like the MHSA and FFN <ref:2512.05134#pg1>, and skipping computation depends on whether those rules hold true when we run different models or even slightly altered inputs. What are the limitations they admit regarding this caching approach?

Paper summary: Tom: That's a fair point, Meng. The authors do acknowledge that the method relies on computing a per-timestep, per-layer, and per-module binary cache plan matrix from just a few deterministic runs <ref:2512.05134#pg0>. They also have to use quantile thresholds to set this initial plan, which means they need some calibration data to start off <ref:2512.05134#pg2>.

Jane: So, the method isn't completely autonomous; it needs that small deterministic calibration set D to generate the initial plan C(zero) <ref:2512.05134#pg2>. This initial plan is then refined in a second phase using a resampling correction to handle situations where consecutive caches happen, which helps avoid drift <ref:2512.05134#pg0>.

Lu: The cross-scale invariance they target is what makes this interesting; they aren't just caching at the layer level within one step, but they are also deciding at a step level whether an entire step can reuse cached results <ref:2512.05134#pg0>. That ability to plan both across timesteps and layers is what allows for that potential acceleration.

Tom: So, we're talking about a system where the AI first plans *when* to cache an entire step, and then if it does, it plans *which* specific modules within that step are safe to reuse <ref:2512.05134#pg2>. It’s a sophisticated scheduling mechanism built on those feature change statistics we discussed earlier.

Meng: Thinking about the practical impact on deployment, how much does this actually reduce the computational load? The paper mentions a speedup of two point eight six times on DiT-XL/two with fifty steps <ref:2512.05134#pg0>, which is substantial if true, but we need to see that translate reliably across different hardware setups.

Lalam: If we can achieve that kind of reduction in FLOPs, it means the generative capability becomes accessible on more accessible hardware, which is a huge cultural shift for how we deploy advanced generative systems <ref:2512.05134#pg0>. It democratizes access to high-fidelity content creation.

Jane: The paper shows that on DiT-XL/two they see a speedup of two point eight six times and a reduction in FLOPs from twenty-two point eight nine down to seven point nine six <ref:2512.05134#pg0>. That's a significant reduction in the work the hardware has to do per sample, which is what makes it appealing for engineering teams like Meng’s startup <ref:2512.05134#pg0>.

Lu: The fact that they are able to apply this same principle—the invariance criterion—to families like FLUX variants with dual-stream blocks shows the broad applicability of their theoretical framework across different diffusion architectures <ref:2512.05134#pg2>. That structural applicability is what gives this method its real potential.

Tom: It seems the core contribution here is proving that we can derive a training-free caching strategy by identifying these relative feature invariances across both time and layer scales <ref:2512.05134#pg0>. It’s about using statistical observation to guide inference execution instead of relying on exhaustive search methods.

Paper summary: Jane: And the conclusion they draw from this is that their method provides a way to achieve acceleration by explicitly planning reuse at two distinct scales, which is step-level and within a timestep across modules <ref:2512.05134#pg2>. They use that initial plan C(zero) and then refine it with corrections to handle the practical realities of consecutive caching <ref:2512.05134#pg0>.

Meng: So, if we look at the final result, they report up to three point three one times end-to-end acceleration on FLUX <ref:2512.05134#pg0>.one-dev <ref:2512.05134#pg0>. That kind of performance gain is what makes me want to see how quickly we can integrate this into our existing inference pipeline without major rework <ref:2512.05134#pg0>.

Lalam: I think the most profound implication for culture is that this level of efficiency means we can move beyond generating static images or simple videos and toward truly dynamic, highly personalized AI experiences in real time <ref:2512.05134#pg0>. It allows for a much more responsive interaction with the generative engine.

Tom: So, to wrap up this discussion on "Fine-Grained Caching for Diffusion Transformers with Few Calibration Conditions," we've seen how they use specific feature change metrics like MSEZ and cosine similarity to create a cache plan <ref:2512.05134#pg1>. They show that by combining step-level and layer-level reuse planning, they can achieve substantial speedups on models like DiT-XL/two <ref:2512.05134#pg0>.

Jane: The paper suggests that this training-free approach, relying on few deterministic calibration conditions to set the initial plan C(zero) and then applying a resampling correction, offers a practical path toward accelerating diffusion generation <ref:2512.05134#pg0>. This moves the focus from complex training regimes to smarter inference scheduling.

Lu: The overall message of "Fine-Grained Caching for Diffusion Transformers with Few Calibration Conditions" is that we can derive a powerful, data-driven caching strategy based on inherent feature invariance without needing extensive retraining <ref:2512.05134#pg0>. This shifts the research focus toward developing principled planning algorithms for iterative models.

Meng: From an engineering standpoint, the paper provides a concrete framework for how to structure this planning process—defining those metrics and thresholds—which gives us a clear starting point if we want to build our own caching solutions <ref:2512.05134#pg0>. It’s not just theory; it's a blueprint for implementation.

Lalam: If this technique proves scalable across more complex AI families, it means the next generation of generative models won't be bottlenecked by their slow sampling process but will instead benefit from these inherent structural properties <ref:2512.05134#pg2>. That’s a vital step for the future trajectory of creative AI.

Tom: That's a fantastic summary, folks. We've looked at how InvarDiff tackles the slow nature of diffusion by using feature invariance to create a training-free cache plan and demonstrating significant speedups on models like DiT-XL/two and FLUX <ref:2512.05134#pg0>. That covers our look at the paper today.

Conclusion: Tom: So, we've seen how this paper uses specific feature change metrics to create a training-free cache plan for diffusion models across different scales, and now it’s time to talk about what this whole thing actually means.

Jane: I think the title itself really captures the essence of what they’re doing—fine-grained caching using just a few calibration conditions instead of massive retraining. It sounds like they've found a very smart shortcut for making these models run faster during inference without needing to retrain them on every new task, which is quite an interesting concept.

Lu: From my side, I find the idea of deriving this plan from feature invariance across time and layer scales really fascinating because it suggests a deep structural property of how these diffusion generators operate internally. It’s like finding a natural rhythm in the model's computations that we can exploit for efficiency.

Meng: Exploiting that rhythm is cool, Lu, but I still have to ask how this translates to real-world deployment; if we rely on just a few calibration runs, does that mean the performance might vary depending on which specific AI architecture we use?

Lalam: That’s exactly what makes this paper so compelling for the future of AI; it suggests that instead of needing endless training cycles for every new application, we can leverage these inherent structural properties to rapidly adapt and accelerate generation capabilities across different systems.

Tom: Exactly! It’s about moving away from brute-force training towards intelligent inference scheduling based on observing how features move through the network. This isn't just a speed tweak; it’s a fundamental shift in how we think about running complex generative models.

Jane: And when you look at the authors, they clearly have a strong grasp on diffusion architectures and the mathematical underpinnings of these transformation models, which is essential because this method relies heavily on those theoretical concepts to work correctly.

Lu: The methodology itself is impressive; they established a two-phase calibration process with an initial plan followed by a resampling correction to handle drift, which shows a very thorough understanding of the potential pitfalls in cross-scale caching.

Meng: I'm still focused on the practical reality, though; how do we operationalize those quantile thresholds they use? Getting those settings right for different hardware setups sounds like it could be a real engineering hurdle before this becomes something practical for widespread use.

Lalam: That challenge is what makes this paper so important because it sets the stage for developing more robust and adaptive AI systems that can perform efficiently across a wider variety of hardware and tasks without constant, expensive retraining cycles.

Zihao Wu

Peking University

cs.CV, cs.DC, cs.LG

Submitted: 2025-11-29

Updated: 2026-10-03

Code: https://github.com/zihaowu25/InvarDiff

Importance score: 87/100

The gist: InvarDiff presents a training-free, cross-scale caching scheme for DiT-family diffusion generators by exploiting relative feature invariance across timestepscale and layer-scale.

Key concepts

Cache Plan Matrix (C)
This binary matrix specifies exactly which modules at which step should be reused instead of being recomputed. It is derived from deterministic runs using change metrics to decide if a specific layer or module can safely reuse its previous output, enabling both layer-wise and cross-timestep caching.
Temporal Invariance Metrics
These measures quantify how much the model's internal representations change over time (between consecutive timesteps). They include MSEZ for magnitude changes and cosine angle for directional changes in token representations. These metrics are used to determine if features are stable enough to be cached.
Two-Phase Calibration
The plan is estimated without retraining using a small calibration set. Phase 1 establishes an initial plan based on fixed quantile thresholds derived from change rates. Phase 2 refines this by re-running the model under simulated reuse conditions to generate more accurate layer and step gates.
Inference-Time Scheduling
At test time, the fixed plan dictates computation. If a step gate is set to 1, the entire forward pass for that step is skipped, reusing previous outputs. If it's 0, the system selectively reuses or computes specific layers based on the cache plan before updating its stored results.

Terminology

Summary

InvarDiff presents a training-free, cross-scale caching scheme for DiT-family diffusion generators by exploiting relative feature invariance across timestepscale and layer-scale. This method addresses the slow, iterative nature of diffusion models by computing a deterministic cache plan based on feature change statistics and using a re-sampling correction to avoid drift when consecutive caches occur, achieving 2–3× end-to-end speedups with minimal impact on standard quality metrics.

How it works

The method derives a binary cache plan matrix, denoted as C, which specifies which module at which step is reused rather than recomputed. This plan is computed from a few deterministic runs using per-timestep, per-layer, per-module binary cache plan matrix and quantile-based change metrics. The same invariance criterion is applied at the step scale to enable cross-timestep caching, deciding whether an entire step can reuse cached results.

Empirical Invariance and Metrics

Temporal invariance is quantified using two complementary measures for each layer l and submodule s ∈ S = (MHSA, FFN):

  1. MSEZ(s)l,t, Z(s)l,t−1 to capture magnitude/energy changes.

  2. cos∠Z(s)l,t, Z(s)l,t−1 to capture directional changes of token representations.

The layer/module change rate is defined as:

  1. ρ(s)l,t = (Z(s)l,t+1 − Z(s)l,t)(Z(s)l,t − Z(s)l,t−1))−1

This emphasizes stretches where updates shrink over time. A step-level rate is also measured:

  1. ρ(net)t = zt+1 − zt1 / zt − zt−11

Two-Phase Calibration

The plan is estimated from a small deterministic calibration set D without retraining, consisting of a per-step gate c step t and the matrix C ∈ T × L × S.

Phase 1: initial plan.

Run deterministic trajectories on D to record Z(s)l,t and zt for all t and l. Compute ρ(s)l,t and ρ(net)t over D. Choose fixed quantile thresholds (τMHSA, τFFN, τstep) to set the initial binary plan C(0). The first step and the final step are forced to compute.

Phase 2: resampling correction.

Re-run the model without skipping any computation and measure rates under simulated consecutive reuse. The layer-wise correction applies C(0), replacing Z(s)l,t with cached Z(s)l,t−1 when C(0)[t, l, s]=1 to yield refined layer plan C˜. The step-level correction applies c step(0), using z t−1 when c step(0)t = 1 to yield refined step gate c˜step.

Inference-Time Scheduling

At test time, the fixed plan (C, c step) is followed verbatim. For iteration t:

  1. If c step t = 1, skip the entire forward pass at step t and set zt ← zt−1; cached submodule tensors are implicitly valid for this step.

  2. If c step t = 0, execute step t while applying the layer/module plan: reuse the MHSA output when C[t, l, MHSA]=1 and otherwise compute it and overwrite the cache; then apply the same rule to FFN using C[t, l, FFN]. Newly computed outputs always update the cache. After a step-level reuse at t, mask all layer-level caches at t+1 once to avoid stale cross-step chaining.

Adapting to FLUX and DiT-style Variants

For FLUX variants with dual-stream blocks, the caching is applied at the level of families SFLUX = (dual attn, dual ff, dual context ff, single attn, single ff). A shared step gate c step is used for both streams. The final plan C and c step are computed using unified metrics across all families.

Results and Practical Notes

Experiments on DiT-XL/2 show a speedup of 2.86× (Ours fast) with comparable LPIPS/SSIM to Learning-to-Cache, while cutting FLOPs(T) from 22.89 to 7.96. On FLUX.1-dev, the method attains up to 3.31× end-to-end acceleration (Ours fast).

Improvements for AI systems

Based on the provided scientific paper, InvarDiff: Cross-Scale Invariance Caching for Accelerated Diffusion Models, here are specific improvements that can be made to AI systems, along with what those improved systems will be capable of:

The core improvement is the introduction of a training-free, deterministic caching policy that exploits feature invariance across both temporal (timestep) and spatial (layer/module) scales in diffusion models. This allows for significant inference acceleration without sacrificing visual fidelity.

Here are the specific improvements and capabilities:

  1. The system can perform high-fidelity image and video generation using a reduced number of sampling steps while maintaining near-full quality, achieving speedups of up to 3.31× on FLUX and 2.86× on DiT variants compared to full computation.

  2. The AI system can generate complex, high-resolution images (e.g., 1024x1024) and videos with reduced latency, making interactive content creation much faster for users and enabling real-time editing applications that currently struggle with the computational cost of iterative denoising.

  3. The system can serve high-fidelity generative models at scale (e.g., in cloud environments or on resource-limited hardware like the RTX 4070S), drastically reducing inference costs and energy consumption due to lower FLOPs per sample (e.g., cutting DiT FLOPs(T) from 22.89 to 7.96 for the fast mode).

  4. The system can generate diverse content (across different prompts, classes, and seeds) with a generalized cache plan that transfers effectively across unseen data, ensuring stable quality even when generalizing to new inputs or styles without requiring extensive retraining or fine-tuning for each new task.

  5. The AI system can be deployed in production environments where it benefits from being orthogonal to other optimization techniques (like step reduction or quantization), allowing it to compose with existing efficiency gains for compounded speedups, resulting in robust performance across various hardware setups and model architectures (DiT, FLUX).

In essence, the improved AI system will be a generation engine that is simultaneously:

  1. Faster (lower latency).

  2. More Efficient (lower compute/cost).

  3. High-Quality (comparable to full computation), across a wide and tunable range of speed-quality trade-offs.

Sources

Related papers