Understanding the Energy Scaling of Large Language Model Inference Across Context Lengths and Attention Architectures

arXiv:2608.25096 · cs.LG, cs.AI · Submitted 2026-08-25 · Read on arXiv

cs.LG, cs.AI

Submitted: 2026-08-25

Updated: 2026-08-25

Comments: 8 pages, 7 figures

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Decode — 55× in this paper · explained in 2 episodes
• Attention Mechanism — 32× in this paper · explained in 3 episodes
• KV cache — 13× in this paper · explained in 3 episodes
• Transformer — 10× in this paper · explained in 2 episodes
• Benchmarking — 6× in this paper · explained in 3 episodes
• Grouped Query Attention — 6× in this paper
• Energy Efficiency — 5× in this paper · explained in 3 episodes
• Multi-Head Attention — 3× in this paper
• Grouped Query Attention with Sliding Window Attention — 2× in this paper
• Transformer Architecture — 2× in this paper · explained in 2 episodes
• Accountability — 1× in this paper · explained in 2 episodes
• Accuracy — 1× in this paper · explained in 1 episode
• Consistency — 1× in this paper · explained in 3 episodes
• Edge Deployment — 1× in this paper · explained in 2 episodes
• Generative Model — 1× in this paper · explained in 2 episodes
• Generative Models — 1× in this paper · explained in 2 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Quantization — 1× in this paper · explained in 12 episodes

Sources

• Language Models are Few-Shot Learners
• Power Hungry Processing: Watts Driving the Cost of AI Deployment?
• Sustainable AI: Environmental Implications, Challenges and Opportunities
• Carbon Emissions and Large Neural Network Training
• Attention Is All You Need
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• Longformer: The Long-Document Transformer
• From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference
• The Price of Prompting: Profiling Energy Use in Large Language Models Inference
• Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inference
• TokenPowerBench: Benchmarking the Power Consumption of LLM Inference
• Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models
• Mistral 7B
• BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
• Where Do the Joules Go? Diagnosing Inference Energy Consumption
• OPT: Open Pre-trained Transformer Language Models
• Gemma 2: Improving Open Language Models at a Practical Size
• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

Related papers

• Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
• AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
• Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
• Convergence issues in Relational Concept Analysis based on AOC-posets
• Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
• Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks