Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures

arXiv:2609.25624 · cs.AR, cs.LG · Submitted 2026-09-22 · Read on arXiv

cs.AR, cs.LG

Submitted: 2026-09-22

Updated: 2026-09-22

Comments: 14 pages, 5 figures

Code: https://github.com/lpc97667/rf

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Decode — 38× in this paper · explained in 2 episodes
• Core — 32× in this paper · explained in 2 episodes
• Benchmark — 17× in this paper · explained in 2 episodes
• Ablation — 11× in this paper · explained in 1 episode
• Accuracy — 5× in this paper · explained in 1 episode
• Embeddings — 4× in this paper · explained in 2 episodes
• Quantization — 3× in this paper · explained in 12 episodes
• KV cache — 2× in this paper · explained in 3 episodes
• Large Language Model (LLM) — 2× in this paper · explained in 3 episodes
• Verification — 2× in this paper · explained in 2 episodes
• Ablation study — 1× in this paper · explained in 1 episode
• Alignment — 1× in this paper · explained in 1 episode
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Trace — 1× in this paper · explained in 2 episodes
• Verifier — 1× in this paper · explained in 2 episodes

Sources

• Non-Determinism of "Deterministic" LLM Settings
• Training Verifiers to Solve Math Word Problems
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• LLM-42: Enabling Determinism in LLM Inference with Verified Speculation
• Replayable Financial Agents: A Determinism-Faithfulness Assurance Harness for Tool-Using LLM Agents
• Qwen3 Technical Report
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• Impacts of floating-point non-associativity on reproducibility for HPC and deep learning applications
• An SMT Formalization of Mixed-Precision Matrix Multiplication: Modeling Three Generations of Tensor Cores
• Revealing Floating-Point Accumulation Orders in Software/Hardware Implementations

Related papers

• WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization
• Golden Ruler: A Numeric Format Catalog with Bit-Exact Conformance Vectors for FP8, BF16, MXFP4, and Microscaling Formats
• PoisonCap: Efficient Hierarchical Temporal Safety for CHERI
• Provisioning to Runtime Optimization of a 100 MW-Scale AI Cluster
• Bit-Accurate Modeling of GPU Matrix Multiply-Accumulate Units: Demystifying Numerical Discrepancy and Accuracy
• Optimizing Polynomial Multiplication and Fixed-Weight Sampling for HQC on ARM Cortex-M4