A Survey of Transformer-based Language Models with Focus on Efficiency
cs.CL, cs.AI
Submitted: 2024-05-15
Updated: 2026-09-01
Terminology
Sources
- GPT-4 Technical Report
- LLaMA: Open and Efficient Foundation Language Models
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- Gemma 3 Technical Report
- Competitive Programming with Large Reasoning Models
- Phi-4 Technical Report
- Mixtral of Experts
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- DeepSeek-V3 Technical Report
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Carbon Emissions and Large Neural Network Training
- The Efficiency Spectrum of Large Language Models: An Algorithmic Survey
- A Survey on Efficient Inference for Large Language Models
- Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
- Neural Turing Machines
- Fast Transformer Decoding: One Write-Head is All You Need
- LaMDA: Language Models for Dialog Applications
- Scaling Laws for Neural Language Models
- Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering