Adaptive Spectral Feature Forecasting for Diffusion Sampling Acceleration

arXiv:2603.01623 · cs.CV, cs.LG · Submitted 2026-03-02 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Adaptive Spectral Feature Forecasting for Diffusion Sampling Acceleration".

Tom: Diffusion models are powerful tools for high-fidelity image and video generation, but their iterative sampling process is computationally expensive, posing a bottleneck for real-time applications.

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So, we're diving into this paper, "Adaptive Spectral Feature Forecasting for Diffusion Sampling Acceleration," which sounds like it’s tackling one of the biggest bottlenecks in generating high-quality images and videos using diffusion models. Jane and I were just looking at the title and authors, Jiaqi Han et al., and it immediately tells us they are focusing on how to make those iterative sampling processes much faster.

Jane: It certainly does sound focused on efficiency, Tom; when you hear "Diffusion Sampling Acceleration," it makes me think about those long, slow generation times we’ve all seen with these models. The authors are proposing a way to speed up that part of the process without sacrificing the quality of what we create.

Lu: This paper is interesting because they’re moving away from local approximations and looking at a global approach using the spectral domain, which suggests they're trying to capture dependencies across the entire sequence of diffusion steps at once.

Meng: From an engineering standpoint, that sounds ambitious; if you can skip many expensive denoiser evaluations, that translates directly into faster throughput for deployment. I wonder how much overhead their forecasting mechanism itself adds to the total inference time.

Lalam: I'm really intrigued by the idea of a global forecasting method because it suggests a more holistic way of understanding how features evolve over time during generation, which could fundamentally improve how we structure these generative processes in the future.

Tom: Exactly; and what they propose is that this spectral approach allows them to reuse features globally across long temporal distances, which is something local methods just can't handle well. That’s a big conceptual leap from what we've seen before.

Jane: And the core idea seems to be approximating those latent features as functions over time using Chebyshev polynomials, which is a specific mathematical tool they are using for this global view. It sounds like a sophisticated way to model the temporal dynamics of the diffusion trajectory.

Lu: The motivation they give is that these methods have "well-conditioned recurrences and favorable error behavior at long horizons," suggesting they've done some solid theoretical groundwork to ensure their approximation doesn't just work well locally but maintains accuracy over many steps.

Meng: So, if I understand correctly, instead of just copying the last feature or doing a short expansion, they are fitting these Chebyshev functions online using a ridge regression objective during sampling to predict features at skipped steps. That’s how the actual acceleration happens.

Lalam: That dynamic fitting aspect is compelling because it means the system can adapt its spectral basis representation as it generates something specific, which should lead to better performance on diverse inputs compared to a fixed method.

Title and authors: Tom: Right, and that dynamic fitting using ridge regression is what makes this approach training-free and flexible; they're solving for those coefficients online while sampling. It really moves the complexity from training into the inference phase in a very smart way.

Jane: So, to put it simply, they take a sequence of features over time and approximate them using Chebyshev polynomials so they can predict what those features will look like at steps where we skip the actual network computation. Does that make sense?

Lu: It’s essentially modeling the feature evolution not just as a step-by-step change but as a continuous function over the diffusion timeline, which is powerful for understanding long-range dependencies in the data.

Meng: I’m still thinking about implementation complexity; fitting those coefficients via ridge regression online adds computational steps to every forward pass during sampling that we need to account for.

Lalam: But if that adds a small cost per step and buys us a massive speedup overall, it seems like a worthwhile trade-off for the final output quality.

Tom: That’s the crux of it—the trade-off between computational cost during inference and the significant speedup they're achieving, which the paper claims can be up to four point seven nine times on FLUX.one or four point six seven times on Wan2 point 1-14B without noticeable quality loss.

Jane: And they back that up with empirical results showing that even with these large speedups, the sample quality stays comparable to or even better than existing methods, which is a crucial point for any practical application of this work.

Lu: The error analysis they present, specifically Theorem three point two about the universality of Chebyshev polynomials regardless of the forecasting step size, gives a strong theoretical guarantee that their approximation bounds don't get worse as we try to skip more steps.

Meng: That theoretical robustness is what reassures me; I need methods that are predictable in terms of error accumulation during long generation sequences, not something that degrades unpredictably after a few skips.

Lalam: I think the implication for our culture is that if we can generate these complex visuals much faster and with reliable quality, it opens up possibilities for interactive creative tools and immersive experiences that were previously too slow to realize.

Tom: Absolutely; so we’ve covered the basics of how they use spectral forecasting with Chebyshev polynomials to achieve substantial speedups on models like FLUX.one and Wan2 point 1-14B, and the theoretical backing is pretty solid for long-range performance. Next up, let's look at exactly what specific improvements the authors suggest to make this framework even more practical for real-world use.

Title and authors: Jane: That’s right; understanding how they refine the methodology after their initial proposal is just as important as understanding the core idea behind it.

Lu: The paper details several ways they’ve thought about refining their approach, focusing on making it more adaptive and robust in practice.

Meng: I'm curious about those specific improvements because from my side, I need to know if these refinements translate into a simpler implementation or just more complex tuning parameters for the user.

Lalam: I hope these refinements make the final system easier to integrate into existing pipelines, rather than adding a whole new layer of intricate configuration.

Tom: Alright, let’s break down those suggested improvements in detail so we can see how they stack up against current methods and what they actually mean for deployment.

Jane: We’ll go through the suggested modifications to the Adaptive Spectral Feature Forecasting for Diffusion Acceleration paper now and discuss their practical implications.

Lu: The first set of suggestions focuses on making the forecasting mechanism dynamic, specifically by using ridge regression to fit coefficients online during sampling, which lets it adapt based on what features it sees in that specific generation run.

Meng: That sounds like a significant operational change; fitting something online during the loop means we're adding optimization steps to the inference path itself, which is something I need to model closely.

Lalam: The ability to adapt its representation based on the input features feels very powerful, because it means we aren't relying on one static model of feature evolution for everything.

Tom: And that leads into another point where they suggest an adaptive timestep scheduler, which balances computation by deciding when to use the full network pass versus when to rely on the spectral forecaster based on how far along we are in the generation process.

Jane: That sounds like a smart control mechanism; it prevents error accumulation early on while letting us leverage the speedup later in the process when approximation is most effective.

Lu: The scheduling idea combined with their analysis of long-horizon behavior suggests they’ve thought carefully about where to place the computational load for maximum benefit within the overall sampling trajectory.

Meng: So, it’s not just about skipping steps; it's about intelligently deciding *when* to skip them based on the current state of generation, which is a more nuanced engineering problem.

Lalam: That adaptability could allow us to tailor the speedup level precisely for different use cases, perhaps prioritizing speed for quick prototypes and quality assurance for final outputs.

Tom: Then there’s a specific implementation detail they propose—a "Last Block Only" caching strategy, which suggests focusing the spectral forecasting mechanism specifically on the output of the final attention block instead of every layer's output.

Title and authors: Jane: That makes sense from a memory perspective; if we only cache features from the most recent block, we drastically reduce the memory overhead associated with storing intermediate representations across all layers.

Lu: By restricting that caching focus, they’re trying to keep the spectral forecasting mechanism streamlined and computationally efficient without sacrificing the necessary context for accurate long-range prediction.

Meng: That sounds like a very pragmatic engineering decision; reducing memory footprint while maintaining sufficient data fidelity for the predictor is something I can work with.

Lalam: If we can manage that trade-off well, it means we can deploy these fast models on hardware with more constraints, which is a huge practical win for accessibility.

Tom: And finally, they suggest integrating a regularization weight tuning mechanism based on desired speed and quality metrics like PSNR or LPIPS to dynamically adjust the regularization parameter lambda.

Jane: Tuning that lambda parameter dynamically sounds like an intelligent way to manage the trade-off between how fast we want the generation to be and how good we need the resulting image or video to look.

Lu: That’s a sophisticated control loop; it means the system can self-regulate its approximation level based on real-time quality feedback rather than relying on a fixed setting.

Meng: If we can automate that tuning, it removes a lot of manual guesswork from deployment, which is exactly what we need for reliable production systems.

Lalam: I feel like this whole approach moves us toward systems that can be tuned not just for speed, but for desired aesthetic or fidelity targets automatically, which is really exciting for future creative AI applications.

Tom: So we’ve seen the core concept of using spectral forecasting with Chebyshev polynomials, and now we’re seeing how the authors suggest making it more adaptive through online fitting, dynamic scheduling, targeted caching, and adaptive regularization tuning.

Jane: It seems they've built a very comprehensive framework for accelerating diffusion sampling by addressing both the mathematical modeling and the practical operational hurdles of deployment.

Lu: The paper shows a clear path from a theoretical concept in the spectral domain to a more nuanced, practically configurable system that handles long-term dependencies robustly.

Meng: It’s definitely moving this technology closer to being deployable, but I still need confirmation on how smoothly that online fitting integrates into the main sampling pipeline without introducing too much latency.

Lalam: And ultimately, the successful implementation of the Adaptive Spectral Feature Forecasting for Diffusion Sampling Acceleration paper means we can start generating high-fidelity content in ways that are genuinely interactive and responsive to user needs.

The paper's summary: Tom: So, we've seen how they use Chebyshev polynomials to model features over time to speed up diffusion sampling, but now we need to really break down what their summary actually says about this approach and why it matters.

Jane: That's right; they essentially boiled the whole method down into a few key points that explain how this spectral forecasting works and where the major wins are coming from.

Lu: The core takeaway is that they’re moving past just looking at immediate neighbors in time; instead, they’re approximating those features as continuous functions over the entire diffusion trajectory using these special mathematical tools.

Meng: That's a good way to put it; so it’s about capturing the long-range temporal dynamics of the features, which is where local methods usually fall short.

Lalam: And what I find particularly important is their claim that this doesn't just speed things up; they show that the quality stays high, even when you are skipping many steps in the generation process.

Tom: Exactly! It’s not just about getting a faster output; it’s about maintaining or even improving the fidelity of what we create while making the entire process much quicker.

Jane: They emphasize that this is a training-free method, which means you don't need to retrain your diffusion model just to get this acceleration feature working in your pipeline.

Lu: That’s a huge practical advantage; it keeps the deployment simple and avoids the massive computational cost of retraining large models for every new speed optimization.

Meng: From an engineering standpoint, being training-free is fantastic because it means we don't have a whole new training infrastructure to manage just for this acceleration layer.

Lalam: It really changes how we think about deploying these creative tools; instead of spending weeks fine-tuning models for every specific use case, we can deploy powerful capabilities and then tune the inference settings on the fly.

Tom: And those empirical results they cite, showing speedups up to nearly five times on models like FLUX.one really underscore how effective this spectral approach is in practice.

Jane: It’s a lot of work for them to get that level of performance without sacrificing quality during the process, so when you see those numbers, it tells you they did their homework on the error bounds.

Lu: Their theoretical proof that these Chebyshev polynomials provide uniform approximation bounds regardless of how large the forecasting step is really gives confidence in its robustness across different speedup scenarios.

Meng: That theoretical guarantee is what makes me lean towards this; I need a method that won't suddenly become unstable or wildly inaccurate when I push the system to its limits with high speedups.

Lalam: And for our culture, this points toward a future where creative AI isn't just about massive models; it’s about intelligent, adaptive sampling systems that can deliver high-quality results reliably and quickly.

Tom: So, while they tackle the math of spectral forecasting and online fitting during sampling to achieve massive speedups, the real win is showing that this adaptation doesn't come at a cost to image quality.

Jane: Precisely; it’s a way for us to get closer to real-time generation for high-fidelity content without having to accept a significant drop in visual quality.

Lu: This paper opens up so many possibilities for how we can structure generative pipelines, moving away from purely sequential, step-by-step processing toward more holistic temporal modeling.

Meng: I'm still thinking about the operational hurdle of that online fitting—how smooth is the integration into a production inference engine? That’s where I need to know if it adds unacceptable latency.

Lalam: And looking ahead, this technology suggests we can build truly responsive creative tools where user input translates into high-fidelity visuals with minimal wait time.

Tom: We've seen the core mechanism and the impressive speed numbers; now we're seeing how these authors are refining it to make it a practical, robust tool for real-world deployment.

The paper's improvements: Tom: Okay, so we've talked about how they use Chebyshev polynomials for global feature forecasting, and now we're looking at the specific ways they’ve tweaked this method to make it even more useful for real-world systems.

Jane: That’s right; the authors laid out a set of improvements that focus on making the system more dynamic and less rigid during its operation.

Lu: One big refinement they propose is integrating an adaptive timestep scheduler, which means the system can intelligently decide when to use the full network pass versus when to rely on that spectral forecaster based on where it is in the generation.

Meng: A dynamic scheduler sounds like a lot of control logic; from an engineering standpoint, I need to know how smoothly that transition happens between those two modes without introducing noticeable latency during inference.

Tom: That’s the core idea—balancing computation so you don't waste resources early on when you don't need the speed, but you get the massive acceleration later when skipping steps is most effective.

Jane: They also suggest a "Last Block Only" caching strategy, which is a smart move for managing memory because it limits where they keep those intermediate features in mind.

Lu: That’s an engineering choice that keeps the overall memory footprint low while still providing enough context for the spectral forecaster to make accurate predictions without needing to store every single layer's output.

Meng: Reducing memory usage is always a win when you're deploying on constrained hardware, so that’s a practical benefit I can definitely get behind.

Tom: And finally, they suggest tuning the regularization parameter lambda based on real-time quality metrics like PSNR or LPIPS, which lets the system adjust its approximation level automatically.

Jane: That dynamic tuning is what makes this approach really sophisticated; it means the system can self-regulate its trade-off between speed and fidelity depending on what you actually want to see in the final output.

Lu: I think that self-regulation capability, combined with the online fitting during sampling, creates a highly personalized acceleration engine for every single generation run.

Meng: If we can automate that quality feedback loop, it removes a lot of manual tuning guesswork from our production pipeline, which is something I really need for reliable deployment.

Tom: So these refinements show the authors aren't just proposing a static tool; they’re building a system that can evolve and adapt to the specific needs of the image or video being generated at any given moment.

Jane: It’s like giving our creative AI a kind of internal feedback mechanism so it knows exactly how much approximation is needed for the best result.

Lu: This whole suite of improvements suggests that this framework isn't just about a speed boost; it’s about building a more flexible and controllable foundation for next-generation generative workflows.

Meng: I need to see those integration steps clearly so I can estimate how much refactoring is required on our existing diffusion pipelines to adopt these changes.

Tom: We've covered the core concept, the results, and now we're digging into the practical tuning knobs they’ve added—it’s clear this paper is laying down a very thorough blueprint for making diffusion models truly usable in fast workflows.

Conclusion: Tom: So we've covered how they use spectral forecasting with Chebyshev polynomials to model features over time for diffusion sampling acceleration, and now we’re wrapping up with a look at what this means for the future of generative AI.

Jane: That's right; essentially, these authors have shown a way to significantly speed up those iterative sampling processes by using global feature prediction instead of local approximations.

Lu: The implications are huge because it suggests we can move toward more holistic temporal modeling in diffusion generation, allowing us to handle very long sequences with much less computational strain.

Meng: From an engineering standpoint, this points toward a future where high-fidelity video and image generation isn't just a slow process anymore, but something that can be deployed efficiently in environments that require rapid response.

Lalam: For our culture, this means we can build creative tools that are truly responsive, allowing users to create complex visuals with minimal wait time or computational overhead.

Tom: And the practical impact is clear: we're looking at potential speedups of nearly five times on models like FLUX.one and significant gains on Wan2 point one-14B without sacrificing the visual quality we expect from these models.

Jane: It really demonstrates that you can achieve substantial performance boosts by focusing on the mathematical structure of the features rather than just brute-forcing more network evaluations.

Lu: The theoretical robustness they proved with Theorem three point two about Chebyshev polynomials being universally applicable regardless of step size is a powerful statement for researchers in this area.

Meng: I still have a question about the online fitting process; how stable is that adaptation during high-speedups where the system is under extreme pressure?

Tom: That's fair, Meng; that dynamic adjustment mechanism needs to be really solid when you push it hard.

Jane: Overall, this paper on Adaptive Spectral Feature Forecasting for Diffusion Sampling Acceleration gives us a very clear path toward building faster, more reliable generative systems.

Lu: This work is foundational for thinking about how we model complex dependencies across sequential data in generative tasks.

Meng: I think the integration of that adaptive scheduler and the targeted caching strategy are what will determine its success in a production environment.

Lalam: Ultimately, this research paves the way for interactive creative AI where latency isn't a barrier to expression.

Jiaqi Han, Juntong Shi, Puheng Li, Haotian Ye, Qiushan Guo, Stefano Ermon

Stanford University · ByteDance

cs.CV, cs.LG

Submitted: 2026-03-02

Updated: 2026-09-30

Comments: CVPR 2026

Code: https://github.com/black-forest-labs/flux

Project page: https://hanjq17.github.io/Spectrum

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 85/100

The gist: Diffusion models are powerful tools for high-fidelity image and video generation, but their iterative sampling process is computationally expensive, posing a bottleneck for real-time applications.

Key concepts

Chebyshev Polynomials: These are special mathematical functions used to approximate complex shapes or data points efficiently. In this paper, they are used to model how a feature changes over the diffusion process (time), providing a compact way to represent these temporal dynamics.
Spectral Domain Approach: Instead of looking at features step-by-step, this method views each feature channel as a function of time. It then approximates these functions using global, orthonormal bases in the spectral domain, which makes capturing long-range dependencies much more effective than local methods.
Online Fitting and Forecasting: Spectrum fits the coefficients for the polynomial approximation continuously during sampling (online fitting). Once fitted, it uses these cached coefficients to predict features at subsequent steps without needing to run the expensive denoiser network, thus skipping many costly evaluations.

Terminology

Chebyshev Polynomials — 37× in this paper · explained in 2 episodes
Diffusion Model — 37× in this paper · explained in 8 episodes
Diffusion Models — 31× in this paper · explained in 6 episodes
Transformer — 14× in this paper · explained in 2 episodes
Core — 13× in this paper · explained in 2 episodes
Fidelity — 13× in this paper · explained in 2 episodes
Ablation — 12× in this paper · explained in 1 episode
Regularization — 11× in this paper · explained in 3 episodes
Benchmark — 10× in this paper · explained in 2 episodes
Consistency — 7× in this paper · explained in 3 episodes
Distillation — 7× in this paper · explained in 2 episodes
Ablation study — 6× in this paper · explained in 1 episode
Pruning — 6× in this paper · explained in 4 episodes
Robustness — 6× in this paper · explained in 5 episodes
Generative Model — 4× in this paper · explained in 2 episodes
Generative Modeling — 4× in this paper · explained in 3 episodes
Generative Models — 4× in this paper · explained in 2 episodes
Approximation Bound — 3× in this paper · explained in 1 episode
U-net architecture — 3× in this paper · explained in 3 episodes
Ablation Studies — 2× in this paper · explained in 1 episode
Accuracy — 2× in this paper · explained in 1 episode
Adaptation — 2× in this paper · explained in 1 episode
Adaptive Sampling — 2× in this paper · explained in 1 episode
Distribution Matching — 2× in this paper · explained in 2 episodes
Flow Matching — 2× in this paper · explained in 7 episodes
Quantization — 2× in this paper · explained in 12 episodes
Rectified Flow — 2× in this paper · explained in 3 episodes
Adaptability — 1× in this paper · explained in 1 episode
Adaptive Scheduler — 1× in this paper · explained in 1 episode
Computational overhead — 1× in this paper · explained in 2 episodes
Fine-Tuning — 1× in this paper · explained in 5 episodes
Flow Matching Model — 1× in this paper · explained in 2 episodes
Generalization — 1× in this paper · explained in 13 episodes
Generative AI — 1× in this paper · explained in 2 episodes
Latent Diffusion Model — 1× in this paper · explained in 2 episodes
Latent Diffusion Models — 1× in this paper · explained in 2 episodes
Redundancy — 1× in this paper · explained in 2 episodes
Score Function — 1× in this paper · explained in 4 episodes
Verification — 1× in this paper · explained in 2 episodes

Summary

Diffusion models are powerful tools for high-fidelity image and video generation, but their iterative sampling process is computationally expensive, posing a bottleneck for real-time applications. This paper introduces Spectral Diffusion Feature Forecaster (Spectrum), a training-free approach that accelerates diffusion sampling by enabling global, long-range feature reuse in the spectral domain. By approximating latent features as functions over time using Chebyshev polynomials, Spectrum achieves significant speedups—up to 4.79× on FLUX.1 and 4.67× on Wan2.1-14B—while maintaining sample quality comparable to or superior to existing methods, which is crucial for streamlining diffusion generation into real-time systems.

The Core Problem and Motivation

Diffusion models rely on numerous iterative passes of a large denoiser (typically a Diffusion Transformer), leading to high inference costs. Previous feature caching schemes, such as naive copying or Taylor expansion, suffer from compounding approximation errors when the forecasting horizon grows, causing severe quality degradation at high speedups. The authors argue that local predictors struggle to capture the long-range temporal dynamics of features along the sampling process and are prone to producing inaccurate semantics.

The Spectral Domain Approach

Spectrum moves beyond local approximations by viewing each feature channel of the denoiser output along the diffusion trajectory as a function over time, which is then approximated with a compact set of global, orthonormal bases in function space. Specifically:

They leverage Chebyshev polynomials [31] to fit these functions via a ridge regression objective.
This approach is motivated by their well-conditioned recurrences and favorable error behavior at long horizons [31].
The theoretical insight is that this method admits more favorable long-horizon behavior and yields an error bound that does not compound with the step size.

The Spectral Diffusion Feature Forecaster (Spectrum) Framework

The Spectrum framework operates in an online fitting-then-forecasting manner, structured as follows:

The latent features are viewed as functions of time: ht = [h1(t), · · ·, hF (t)] ∈ R F [7].
Each feature function is approximated using Chebyshev polynomials up to degree M: hi(t) = X M sum m=0 c m,i T m(τ) [8], where the projection maps the diffusion timestep to the support of Chebyshev polynomials: "τ = g(t): [0, 1] -> [-1, 1] = 2t - 1" [8].
Coefficients are fitted online during sampling by solving a ridge regression objective (Eq. 12): Ctj = arg min C ΦtjC − Htj 2 F + λC 2 F [12], where Ctj is the feature cache at time tj, and Htj is the stacked cached features.
For steps in V (where no actual network evaluation occurs), the feature is predicted using Eq. 14: htj = ϕ(g(tj))Ctj, thereby skipping expensive denoiser evaluations.

Error Analysis and Robustness

The key theoretical contribution is the error analysis, which demonstrates the superior robustness of Spectrum compared to local predictors like TaylorSeer.

Theorem 3.2 establishes that Chebyshev polynomials provide uniform approximation bounds regardless of the forecasting step size: Theorem 3.2 (Universality of Chebyshev Polynomials).
Theorem 3.3 provides a tight error bound for Spectrum, showing that the error does not scale with the skip horizon: hi(tj) − bhi(tj) ≤ εM(1 + (M + 1)K/σmin(Φj) squared + λ + λ√M+1/σmin(Φj) squared + λ·2B/(p 1−ρ-2), where M is the degree of the basis.
This analysis confirms that Spectrum remains robust across all speedups, achieving much higher quality while using fewer network evaluations.

Empirical Results and Extensions

Extensive experiments on state-of-the-art models (FLUX.1, Wan2.1-14B) consistently verify the superiority of the approach.

: Spectrum achieves up to 4.79× speedup on FLUX.1 and 3.40 to 4.67× speedup on Wan2.1-14B without notable quality degradation [Abstract]. (Table 1, Table 2).

**: Ablation studies show that the degree of Chebyshev polynomials M is key; increasing M from 2 to 4 promotes sample quality, while further scaling up to M=6 brings marginal improvement [Figure 6].

Improvements for AI systems

As a fastidious and diligent AI researcher, I have analyzed this paper, Adaptive Spectral Feature Forecasting for Diffusion Sampling Acceleration (Spectrum). The core innovation lies in replacing local Taylor expansion approximations with a global spectral forecasting method based on Chebyshev polynomials to predict latent features at skipped diffusion steps.

Here are the specific improvements you can make to existing diffusion sampling systems and what the resulting AI system can achieve:

)

Implement a training-free acceleration module for any existing high-fidelity image/video diffusion model (e.g., FLUX, Stable Diffusion, Wan2.1). This module must be integrated into the inference pipeline to skip network evaluations at selected diffusion steps using the Spectrum forecaster instead of naive copying or local Taylor expansion.

Leverage Chebyshev polynomial fitting via ridge regression (Eq. 12) as a dynamic, online coefficient estimation mechanism within the sampling loop (Alg. 1). This allows the system to adapt its spectral basis representation to the specific features encountered during that particular generation run, enhancing robustness over fixed-basis methods.

Adopt an adaptive timestep scheduler (U set defined by Eq. B.1) that dynamically balances computation: performing full network passes at early steps (warm-up phase, W steps) and gradually shifting to the spectral forecaster as the sampling progresses toward the final denoising step. This prevents error accumulation in later stages while maximizing speedup where approximation is most effective.

Design a Last Block Only caching strategy for Spectrum implementation (Section 3.4, Table 4). Instead of caching features from every attention block, focus the spectral forecasting mechanism on the output of the final attention block. This reduces memory overhead by avoiding layer-wise caching while maintaining high sample quality and speedup performance.

Integrate a regularization weight tuning mechanism (Section 4.3, Table 3). Dynamically adjust the regularization parameter λ based on the desired acceleration ratio and current quality metrics (e.g., PSNR/LPIPS). This prevents underfitting (too high λ) or numerical instability (too low λ) during high-speedups, ensuring optimal trade-off between speed and fidelity.

Deploy the resulting accelerated system to achieve up to 4.79× speedup on models like FLUX.1 and 4.67× on Wan2.1-14B without measurable quality degradation (as demonstrated in Table 2). This allows for real-time or near real-time generation of high-fidelity images and complex videos that were previously computationally infeasible due to the iterative nature of Diffusion Transformers.

Abstract

Diffusion models have become the dominant tool for high-fidelity image and video generation, yet are critically bottlenecked by their inference speed due to the numerous iterative passes of Diffusion Transformers. To reduce the exhaustive compute, recent works resort to the feature caching and reusing scheme that skips network evaluations at selected diffusion steps by using cached features in previous steps. However, their preliminary design solely relies on local approximation, causing errors to grow rapidly with large skips and leading to degraded sample quality at high speedups. In this work, we propose spectral diffusion feature forecaster (Spectrum), a training-free approach that enables global, long-range feature reuse with tightly controlled error. In particular, we view the latent features of the denoiser as functions over time and approximate them with Chebyshev polynomials. Specifically, we fit the coefficient for each basis via ridge regression, which is then leveraged to forecast features at multiple future diffusion steps. We theoretically reveal that our approach admits more favorable long-horizon behavior and yields an error bound that does not compound with the step size. Extensive experiments on various state-of-the-art image and video diffusion models consistently verify the superiority of our approach. Notably, we achieve up to 4.79 times speedup on FLUX.1 and 4.67 times speedup on Wan2.1-14B, while maintaining much higher sample quality compared with the baselines.