FORGE: Procedural Weight Generation for Sparse MoE Inference

summary

The gist

The gist: FORGE introduces a procedural weight generation system for Sparse MoE inference that replaces HBM-resident expert weights with on-chip procedural regeneration, leading to estimated

In short

FORGE replaces storing expert weights in high-bandwidth memory with generating them procedurally during inference. This system uses a Rotor Generation Unit to create FFN weight tiles on-the-fly from rotor parameters instead of loading dense tensors. This results in significant throughput gains and lower latency, trading off some downstream accuracy.

Key concepts

Rotor Generation Unit (RGU): A fixed-function datapath embedded in each GPU that generates FFN weight tiles on-the-fly. It replaces the traditional HBM weight loading path with a dedicated generator, allowing generation to happen without competing for compute resources during Tensor Core execution.
Procedural Weight Generation: The technique of approximating dense expert weights using rotor parameters. Each pre-trained expert is fitted offline to a seed, and at inference, this seed acts as the proxy for the full weight tensor, allowing weights to be synthesized dynamically.
Supertile: A specific tile structure resulting from the intersection of algebraic block-diagonal structure and GEMM tile granularity. It dictates how data is generated so that each Streaming Multiprocessor consumes exactly one weight tile per K-dimension iteration, enabling full pipeline overlap.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 20 episodes
Retrieval-Augmented Generation (RAG) — explained in 18 episodes
Federated Learning — explained in 17 episodes
Quantum Fisher Information (QFI) — explained in 16 episodes
Flow Matching — explained in 15 episodes

This episode discusses

The paper

FORGE: Procedural Weight Generation for Sparse MoE Inference · Read on arXiv

Chanyoung Moon, Travis Pence, Daisuke Yamada, Jiaqi Mo, Yanxin Zhang, Vikas Singh, Karthikeyan Sankaralingam

University of Wisconsin-Madison · NVIDIA

We introduce FORGE (Fast On-chip Rotor Generation for Experts), a procedural weight generation system for Sparse Mixture-of-Experts (MoE) inference. Rather than storing and loading every expert weight tensor from HBM, we show how a compact accelerator inside each Streaming Multiprocessor can regenerate them on-the-fly from O(squared d) rotor parameters per projection. Per replaced layer, gigabytes of stored expert weights are approximated with under a megabyte of seeds. This addresses the dual memory bottleneck of autoregressive Sparse MoE decode, where per-expert FFN operations are HBM-bandwidth-bound at small routed-token counts and the aggregate expert footprint forces multi-node Expert Parallelism (EP) with all-to-all traffic. We realize the approach as the Rotor Generation Unit (RGU), a fixed-function datapath that generates FFN weight tiles without competing with Tensor Core execution. With 30% of the MoE FFN layers replaced across five Sparse MoE models, we estimate 1.3-2.8x higher system throughput at matched GPU count and 1.10-1.15x lower per-user decode latency at short context. In exchange, downstream accuracy on three model families falls below that of the same model fine-tuned without rotor replacement, and the replacement ratio controls this tradeoff. FORGE also expands KV cache capacity by 1.2-2.6x. It is agnostic to the bit width at which weights are stored, so it can compose with quantization. The RGU is fully implemented/verified in SystemVerilog at 0.9% SM area overhead (0.032 mm 2/SM, 7nm).

DOI: 10.1145/3845814.3850033

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "FORGE: Procedural Weight Generation for Sparse MoE Inference".

Jane: The gist: FORGE introduces a procedural weight generation system for Sparse MoE inference that replaces HBM-resident expert weights with on-chip procedural regeneration,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we've seen the setup for FORGE: Procedural Weight Generation for Sparse MoE Inference. We know it’s about replacing HBM weight loading with on-the-fly generation using rotor parameters.

Jane: Right, but what is the main thesis they are pushing? What problem are they trying to solve with this system?

Lu: The main goal is to address the dual memory bottleneck of autoregressive Sparse MoE decode by regenerating expert weights on-the-fly from O(log2 d) rotor parameters per projection instead of storing and loading every expert weight tensor from HBM.

Meng: So, the core claim is that a compact accelerator inside each Streaming Multiprocessor can do this regeneration without competing with Tensor Core execution. It moves the bottleneck away from memory bandwidth to compute bound execution for those replaced layers.

Lalam: They show how to adapt the Clifford-algebra rotor framework from attention layers to MoE FFN decode, and develop a parallel execution model that exploits column, row-pair, and k-block independence with precomputed conflict-free pair scheduling suited for hardware realization.

Tom: And they mention a specific result: per replaced layer, giga accuracy falls below what you get from the same model fine-tuned without rotor replacement.

Jane: That's the quality cost mentioned early on, but they argue that this substitution is tunable. With thirty percent of the MoE FFN layers replaced, accuracy falls below that level, but higher ratios promise larger system gains.

Lu: They show how the capacity advantage of procedural generation widens with model scale because while stored weight volume scales linearly, rotor parameter seeds scale logarithmically.

Meng: So, even for bigger models like Qwen3-30B, the scaling of those parameters is favorable for this approach compared to just storing more dense weights <ref:2610.16890#pg1>.

Lalam: The paper shows that they adapt the Clifford-algebra rotor framework fifty-one from attention layers to MoE FFN decode in their contribution list <ref:2610.16890#pg1>.

Conclusion: Tom: So, wrapping up FORGE: Procedural Weight Generation for Sparse MoE Inference by Moon et al., it’s this system replaces HBM weight loading with procedural on-the-fly generation for Sparse MoE FFN layers.

Jane: In simple terms, the authors are proving that you can use a compact accelerator inside each processing unit to create expert weights where they are needed during inference, rather than having to load them from the slow HBM every single time.

Lu: The implication is that this shifts the execution model for these specific layers from being bandwidth-bound to compute-bound, which we know is what we want when dealing with large sparse models.

Meng: It means you can get substantial throughput gains and reduce latency at short context, provided you are willing to accept a controlled reduction in downstream accuracy relative to a standard fine-tuned model.

Lalam: The paper shows that with thirty percent of the MoE FFN layers replaced, we estimate one point three to two point eight times higher system throughput at matched GPU count and one point ten to one point fifteen times lower per user decode latency at short context, while the KV cache capacity grows by one point two to two point six times.

Tom: That’s what the numbers show on paper, and it’s exciting because those gains are independent of context length for throughput, relying only on freed HBM capacity.

Jane: It seems like a solid way to gain massive efficiency in sparse MoE inference by using clever algebra to replace slow memory access with fast on-chip generation.

More episodes

← Home