One Block, Multiple Depths: Recurrent Vision Transformers with Depth-Programmed Experts

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "One Block, Multiple Depths".

Jane: The gist: reViT restores depth-specific transformations by representing the FFN at each recurrent depth as a convex combination of a small shared expert bank,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Alright, let's talk about the title and who wrote this paper. The paper is called "One Block, Multiple Depths: Recurrent Vision Transformers with Depth-Programmed Experts." It’s a pretty descriptive name that immediately tells you what the main trick is.

Jane: It sounds like it’s focused on using recurrence within a single block to achieve multiple depths of computation without needing to stack many separate blocks. They list Adrian Bulat, Yassine Ouali, and Georgios Tzimiropoulos as the authors.

Lu: I think the authors are tackling a very specific architectural challenge: how to efficiently handle depth-specific transformations in vision models when you want to keep the computational footprint manageable.

Meng: I’m curious if they are proposing something that drastically changes how we think about scaling vision models, or if it's more about optimizing existing structures for better performance at a given cost.

Lalam: It feels like they are focusing on the internal mechanics of a transformer block, trying to make that single block do more work intelligently across different scales.

Tom: They are doing exactly that. They’re not proposing an entirely new type of network structure, but rather showing how a single recurrent Transformer block can be applied recurrently to match the accuracy of a full-depth vision encoder at comparable FLOPs.

Jane: The big implication here is efficiency. If you can achieve full-depth accuracy with fewer FLOPs, it makes models more accessible for real-time applications where computational power is limited.

Lu: It shifts the focus away from just adding more layers, which has been a common approach, toward smarter internal computation patterns within the block itself.

Meng: That means we spend less time just building longer sequential stacks and more time figuring out how to make those individual layers work harder with better parameter utilization.

Lalam: It suggests that the future of efficient vision models might lie in making each fundamental block more versatile, capable of adapting its behavior based on the input context or desired depth.

The paper's summary: Tom: Now, let's get into the actual mechanics described in "One Block, Multiple Depths: Recurrent Vision Transformers with Depth-Programmed Experts." Basically, they show that they represent the FFN at each recurrent depth as a convex combination of a small shared expert bank.

Jane: They use this continuous normalized-depth coordinate to program this mixture, which defines a resampleable trajectory through the FFN parameter space. This is how they define the depth-specific transformations.

Lu: So, at each recurrent step, there’s one dense FFN that is produced by applying this weighted combination of experts to every token in the sequence.

Meng: It seems like the key here is that this single dense FFN is applied universally across all tokens after it's been programmed by that depth coordinate.

Lalam: So, instead of having different specialized layers for each depth, you have one shared structure whose output changes based on where you are along the depth trajectory.

Tom: That’s right. It replaces the independently learned depth-wise transformations with this continuous path through parameter space. This is a departure from static hypernetworks because they don't assign a fixed embedding to each depth level beforehand.

Jane: The router maps that scalar coordinate s t = t / (L - one) to soft coefficients over the expert bank, defining a soft mixture of experts at each recurrent depth <ref:2610.12448#pg1>.

Lu: They evaluate this design in two regimes: supervised ImageNet-1k training and distillation from a DINOv2 teacher <ref:2610.12448#pg1,evaluate this design in two regimes: supervised ImageNet-1k training and distillation>. Across both regimes, controlled adaptations identify weight-space merging as the strongest MoE family tested when you are restricted to one FFN budget.

Meng: So they are finding that weight-space merging is the most effective way to use a small number of experts when you have a tight budget on a single FFN computation.

Lalam: It’s about finding the best way to pack that accuracy into the fewest possible parameters by using this expert bank effectively.

The paper's improvements: Tom: Moving on to how they suggest improving this design, they focus heavily on replacing simpler MoE approaches with their method. They argue that weight-space merging is superior to token-dispatch and output-mixture alternatives when applied to a single recurrent block.

Jane: They also talk about how they found that expanding the expert bank size consistently recovers accuracy lost under plain weight tying, showing that increasing E from four to eight experts improves S/sixteen by two point zero on ImageNet points during supervised training <ref:2610.12448#pg1>.

Lu: So if you are training from scratch, having more experts helps boost the model's performance at larger scales while keeping the computation manageable because they still only evaluate one dense FFN per step.

Meng: That’s interesting because it means we get a direct trade-off: more capacity for better accuracy versus maintaining a lower FLOP count for that single FFN evaluation.

Lalam: It shows that capacity isn't just about throwing more parameters at the problem; it’s about structuring those parameters in a way that allows the model to leverage them more effectively during training.

Tom: They also noted that this depth-programmed merge outperforms the tested token-dispatch and output-mixture alternatives, which they cite Csordás et al., two thousand twenty-four Liu et al <ref:2610.12448#pg2>., two thousand twenty-four Yang et al <ref:2610.12448#pg2>., two thousand twenty-five and Wang et al <ref:2610.12448#pg2>.

Jane: And they also pointed out that once L is chosen, you can compute this map once and reuse it or insert it into a conventional fixed-depth graph with no routing or merging in the input path.

Lu: This flexibility is huge because it means you can choose to store the compact expert bank, or you can materialize the L merged FFNs directly for a fixed graph if you don't want any runtime routing involved.

Meng: So we have this choice: either a highly optimized compact storage or a pre-compiled sequence of dense layers, and both options maintain the same compute budget per depth step.

Conclusion: Tom: We’ve covered a lot about how "One Block, Multiple Depths: Recurrent Vision Transformers with Depth-Programmed Experts" works—how it uses a continuous coordinate to program experts at each recurrent depth. This technique allows for matching full-depth accuracy at comparable FLOPs.

Jane: Essentially, they found that weight-space merging is the strongest MoE family tested when you are restricted to a single FFN budget, and they showed that increasing the expert bank size improves accuracy by two point zero on ImageNet points when supervised training <ref:2610.12448#pg1>.

Lu: The ability to choose between storing experts or materializing merged FFNs for deployment flexibility is a practical aspect that makes this architecture very adaptable to different deployment scenarios.

Meng: The main thing we take away is that this structure allows us to use fewer parameters while maintaining good accuracy, and it’s a solid path toward more efficient vision systems.

Lalam: It’s about finding a way to structure the network so that the capacity you add actually gets utilized without increasing the cost of running it.

Tom: So that's what this paper is all about: reViT, showing how recurrence and depth programming can give us depth-specific computation while maintaining high accuracy at comparable inference FLOPs. We’ll leave you with a paper that shows we can build more efficient vision systems by intelligently structuring the FFN layers.