TuringLLM: Efficiently Scaling Foundation Models Toward Physical AI

arXiv:2608.30567 · cs.AI · Submitted 2026-08-31 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "TuringLLM: Efficiently Scaling Foundation Models Toward Physical AI".

Jane: The paper was written by N/A (Author list not present in the provided excerpt) from FOUNDATION MODEL TEAM and TuringLLM.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Jane: We also have Lu with us today — senior AI researcher at Tsinghua.

Tom: We also have Meng with us today — lead engineer at a mysterious AI startup.

Jane: We also have Lalam with us today — the in-house Large Language Model.

Tom: Alright, let's get started.

Summary: Tom: So, we've got the big picture, but let's look at what the paper actually says about this model. TuringLLM is a 20B-parameter Mixture-of-Experts language model that activates only about two billion parameters per token on average.

Jane: That number is fascinating because it sounds like a massive model, but it' designed to be much more efficient than dense models which are all running their full parameter count for every single word.

Lu: The core idea here is MoE architecture, where the total capacity is high, but only a fraction of the model fires at any given time.

Meng: They call it "Quantile Routing" and that's the key mechanism they’re using to manage which tokens go to which experts.

Lalam: This means that instead of forcing every token through a rigid pipeline, we can have a more flexible and intelligent way of routing information through the model.

Tom: And the results are actually really strong, showing it achieves overall general capability exceeding Qwen3-8B Base while keeping that low activation cost.

Jane: It's not just good performance, though; it also maintains strong long-context performance across different benchmarks like MMLU and RULER.

Lu: The paper is making a specific case that the capacity constraint isn't just a theory, they are demonstrating real-world capability gains.

Meng: I’m particularly interested in the 128K native context length, which is crucial for processing long histories in tasks like autonomous driving.

Lalam: The system isn's designed to forget everything that happened earlier in the conversation or the video feed, which is a huge leap toward reliable AI.

Improvements: Tom: We’ve covered the basic design and summary, but what's actually making this model better than other improvements?

Jane: The hybrid attention architecture is a big one, Tom. It combines "Lightning Attention" for long sequences with occasional full-attention layers to ensure global interactions don' are lost.

Lu: That five:one pattern—five lightning layers followed by one full layer—is mathematically clever because it keeps the computational growth under control as the sequence grows longer.

Meng: And then, they’ apply capacity-constrained routing during prompt prefill, which is a practical deployment improvement to ensure regular execution time for tokens.

Lalam: This means that when we use this AI in a physical system, the time it takes to process the input shouldn't suddenly spike just because one token needed more effort than others.

Tom: That’s exactly what I mean, Lalam. It improves predictability, which is vital for autonomous systems where timing is everything.

Jane: The paper also detailed a progressive three-stage curriculum for pretraining that extends the context from 4K to 128K, ensuring the model learns long-context patterns naturally.

Lu: They are building the capability layer by layer, not just throwing all's data at it and then hoping we get a good result.

Meng: From an engineering standpoint, this progressive training allows us to scale up to 128K without having a massive amount of wasted computation in the early stages of learning.

Lalam: This structured approach helps the AI understand that long-term dependency—that one small detail at the beginning of a long video feed—is important.

Conclusion: Tom: So, we've seen how TuringLLM achieves a balance between capability and efficiency, but let's wrap up our thoughts on what this all means for the world.

Jane: It seems like this paper has provided a very practical path forward for Physical AI foundation models.

Lu: The ability to handle 128K tokens natively, combined with strong performance in knowledge and STEM tasks, is a monumental step.

Meng: And the fact that it uses only about 2B parameters per token makes it an architecture that can actually fit on specialized hardware for real-time use.

Lalam: It suggests that we are moving away from just powerful LLMs toward building truly capable, efficient physical agents.

Tom: We've been talking about the implications of TuringLLM: Efficiently Scaling Foundation Models Toward Physical AI, and I think the message is clear.

Jane: It’s not just a research curiosity; it’ a blueprint for how large-scale models can be used in real-time systems.

Lu: The way we are thinking about scaling MoE needs to be more adaptive, and this model shows us how that adaptation can lead to stable performance.

Meng: I hope this paves the way for a variety of applications, especially those needing consistent low latency for physical interaction.

Lalam: To conclude, it allows us to build an AI that is not only smart but also reliable and efficient in the culture of our automated world.

Conclusion: Tom: So, we’re wrapping up our discussion on “TuringLLM: Efficiently Scaling Foundation Models Toward Physical AI,” and I think we can all agree that this is a genuinely significant piece of work for the industry.

Jane: It really demonstrates that achieving both high capability and low operational cost isn't just a theoretical goal, it’s a practical reality they are now at.

Lu: That’s where the genius of the dynamic quantile routing comes in; we can finally see how to manage massive model capacity while keeping that computational budget tight.

Meng: From an engineering standpoint, I think the commitment to low prefill latency is what makes this so impactful for real-time systems.

Lalam: It’s about building a foundation that allows our physical agents to be more reliable, not just in terms of smart decisions, but in how consistently they execute those decisions over time.

Tom: That stability is exactly the promise here; it avoids that unpredictable "straggler effect" where one complex token ruins the whole thing.

Jane: And I think it’s exciting that we've seen them achieve a level of general reasoning and knowledge comparable to much larger, dense models.

Lu: It’s about proving that we can scale smarter, not just by pushing raw data volume.

Meng: The architecture is designed to run efficiently on actual hardware, which is the ultimate proof of concept for me.

Lalam: I hope this allows us to build a culture where automated systems don't just execute tasks but understand the context and interact with our world in a more thoughtful way.

Tom: It’s certainly an exciting time to be watching these models evolve toward practical, reliable AI.

N/A (Author list not present in the provided excerpt)

FOUNDATION MODEL TEAM · TuringLLM

cs.AI

Submitted: 2026-08-31

Updated: 2026-08-31

Comments: Technical Report; includes supplementary material

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: The paper presents a highly rigorous benchmarking suite designed to measure the operational efficiency of advanced foundation models, specifically targeting architectures that scale toward physical

Key concepts

Mixture-of-Experts (MoE)
MoE is an architecture where a large model has many specialized components. Instead of using all parameters for every token, only a fraction activates. TuringLLM uses 'Quantile Routing' to intelligently manage which experts process each token, keeping the computational cost low.
128K Native Context Length
This refers to the model's ability to handle very long inputs, such as extended conversations or video feeds. TuringLLM supports a native context length of 128,000 tokens. This capability is crucial for ensuring the AI does not forget details from earlier in a sequence.
Hybrid Attention Architecture
The model uses a hybrid attention design combining 'Lightning Attention' for long sequences with occasional full-attention layers. This structure, often following a five-to-one pattern, helps maintain global interactions while controlling computational growth as the sequence length increases.

Terminology

Summary

The paper presents a highly rigorous benchmarking suite designed to measure the operational efficiency of advanced foundation models, specifically targeting architectures that scale toward physical AI applications. By establishing standardized measurement protocols for core computational blocks—including attention mechanisms and Mixture-of-Experts (MoE) modules—the work provides critical insights into real-world inference performance by excluding systemic overheads like initial compilation and shape specialization.

Attention Implementation Details

The benchmarking process employs stringent measurement conditions, utilizing a 100 ms warm-up period followed by a 5,000 ms measurement period for all sequence lengths. The reported latencies are therefore precise measurements of the operational compute time. Attention implementations vary significantly across models:

  • Qwen Baselines: Full-attention layers are configured with attn implementation="sdpa", which on the H800 GPU dispatches to the fused cuDNN native scaled-dot-product attention kernel.

  • Qwen3.5 Architecture: This model uses a hybrid approach, retaining optimized paths for its linear-attention layers. Specifically, the short causal convolution executes through the optimized causal-conv1d kernel, avoiding generic PyTorch fallbacks to maintain high performance during measurement.

  • Turing-20B-A2B: This model utilizes specialized implementations:

  • Lightning Attention layers employ a custom Triton implementation adapted from Flash Linear Attention, featuring a block size of 512 and decay enabled.

  • The four causal full-attention layers are implemented with FlashAttention-2, which directly uses the known query length for rotary-position construction during the prefill path.

Model-Level Latency Aggregation

To estimate the overall model prefill latency (T b prefill(L)), a representative decoder layer type is measured in isolation and then aggregated based on layer multiplicities. The general formula provided for Turing-20B-A2B is:

T b prefill(L) = T dense(L) + 4T routing,linear(L) + 15T nonrouting,linear(L) + 2T routing,full(L) + 2T nonrouting,full(L)

For other models, specific multipliers are applied: for Qwen3-1.7B, the representative layer latency is multiplied by 28; for Qwen3-4B and Qwen3-8B, this factor is 36.

Mixture-of-Experts (MoE) Module Benchmarking

The MoE module latency experiment measures the forward pass of a complete MoE module in isolation, excluding attention and normalization overhead. Measurements are performed on a single NVIDIA H800 GPU using FP16 precision with a batch size of 1. Two distinct expert execution paradigms are compared:

  • Turing MoE (Quantile Routing): This variant employs Quantile Routing with sigmoid router scores, expert-specific routing thresholds, and an expert capacity factor of gamma = 1.25. For an input containing B times L tokens, the maximum number of assignments processed by each routed expert is bounded by C = gamma times L over k. The process involves ranking router scores and gathering these bounded blocks into tensors for batched matrix multiplications.

  • Top-k MoE Baseline (Dropless Top-k): The baseline uses conventional top-k routing but is implemented using a dropless top-k MoE following the fused-expert execution paradigm. Instead of naive per-expert loops, the implementation sorts token–expert assignments by expert and uses fused kernels for the two expert MLP projections, gated activation, routing-weight scaling, and output aggregation. This optimized path successfully supports variable per-expert workloads while avoiding overhead.

Improvements for AI systems

Architectural and Inference Engine Improvements:

  1. Implement Hybridized Attention Kernels for Optimal Layer Performance:

  • Improvement: Design a flexible attention layer module that automatically selects the most efficient kernel based on the specific block type (e.g., Qwen's linear-attention vs. standard full-attention). Specifically, retain the highly optimized, fused causal convolution path (causal-conv1d kernel) for linear attention blocks while using PyTorch SDPA for full attention blocks.

  • System Capability: The resulting system can achieve superior inference latency by maintaining the intended high-performance execution paths of complex models (like Qwen3.5), preventing performance degradation that occurs when generic fallback operators are used instead of specialized, optimized kernels.

  1. Integrate Advanced Quantile Routing with Bounded Expert Execution for MoE:

  • Improvement: Adopt the Turing MoE's Quantile Routing mechanism combined with its capacity-constrained execution model. This involves calculating expert assignments based on ranked scores and strictly limiting the number of assignments processed by each expert (Assignments gamma times L/E), followed by processing these bounded blocks using batched matrix multiplications.

  • System Capability: The AI system can maintain high inference throughput even with extremely large input sequences (up to 256K tokens) and a high number of experts, significantly outperforming traditional top-k MoE baselines because it strictly controls the computational load per expert, guaranteeing predictable latency scaling rather than suffering from variable workload spikes inherent in unconstrained routing.

  1. Develop a Unified, Fused Expert Execution Pipeline for MoE:

  • Improvement: Implement a robust, dropless top-k MoE engine that uses a fused-expert execution paradigm. This involves sorting all token–expert assignments by expert ID and executing the entire projection sequence (first MLP to gated activation to second MLP to routing weight scaling to scatter-add) using a single, fused kernel pass.

  • System Capability: This system can efficiently manage variable per-expert workloads in a single GPU context. It eliminates the overhead associated with naive expert-by-expert looping, providing a highly optimized and predictable inference path suitable for modern LLM engines like vLLM or SGLang, ensuring that throughput scales linearly with sequence length without unexpected computational bottlenecks from kernel dispatch overhead.

  1. Optimize Prefill Latency Estimation via Multiplicative Aggregation:

  • Improvement: For models composed of mixed layer types (e.g., MoE layers interleaved with standard layers), do not rely on sequential summation of individual layer timings. Instead, establish a calibrated system that estimates the total prefill latency (T prefill(L)) using the weighted, multiplicative aggregation formula derived from representative layer types and their known multiplicities (e.g., T prefill = sum (Multiplicity i times T type i(L))).

  • System Capability: This allows for the creation of highly accurate, model-level latency predictions for complex architectures (like Turing-20B-A2B) without needing to time every single layer instance. It provides engineers with a robust, parameterized method to predict overall performance scaling across different model configurations.

  1. Mandate Strict Measurement Protocols for Reproducibility:

  • Improvement: Integrate mandatory, standardized measurement procedures into the inference engine's testing suite:

  • Use FP16 precision exclusively for all core computations.

  • Enforce the use of CUDA-event timing and synchronization (like triton.testing.do bench) with controlled warm-up/measurement periods (e.g., 100ms warm-up, 5,000ms measurement).

  • Implement mandatory cache clearing (L2 cache cleared before every repetition; CUDA allocator cache cleared between sequence lengths) to isolate and measure the true computational throughput of the algorithm, independent of hardware state leakage.

  • System Capability: The resulting AI development platform provides industry-leading, verifiable performance metrics. This mitigates works on my machine syndrome and allows for direct, apples-to-apples benchmarking comparisons between different architectural optimizations or vendor hardware generations with guaranteed timing fidelity.

Abstract

We present Turing-20B-A2B, a 20B-parameter Mixture-of-Experts language model that activates approximately 2B parameters per token, designed for long-context and latency-sensitive physical AI applications. The model adopts Quantile Routing in a dynamic top-k configuration, enabling token-adaptive expert allocation while maintaining balanced expert utilization and a controlled average compute budget. During deployment, we further apply capacity-constrained routing to prompt prefill for more regular and efficient expert execution, while retaining dropless routing during pretraining. Turing-20B-A2B also employs a hybrid attention architecture that combines Lightning Attention with a small number of full-attention layers for efficient long-context modeling. The model is pretrained with a progressive three-stage curriculum and extended to a native context length of 128K through continued pretraining, with further inference-time extension to 512K using YaRN. Despite its compact active-parameter budget, Turing-20B-A2B achieves, at the base-model stage, overall general capability exceeding Qwen3-8B Base and approaching Qwen3.5-9B Base, while maintaining strong long-context performance and favorable prefill-latency scaling. These results demonstrate an effective balance among model capability, long-context scalability, and practical inference efficiency.

Sources

Related papers