What is the Role of Small Models in the LLM Era: A Survey
cs.CL
Submitted: 2024-09-10
Updated: 2026-09-21
Comments: a survey paper of small models
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Large Language Models (LLMs) have demonstrated remarkable capabilities in various reasoning tasks, which leads to the development of increasingly large models.
Terminology
Abstract
Large Language Models (LLMs) have demonstrated remarkable capabilities in various reasoning tasks, which leads to the development of increasingly large models. However, scaling up model sizes results in significantly higher computational costs and energy consumption, which makes these models impractical for academic researchers and businesses with limited resources. At the same time, Small Models (SMs) are frequently used in practical settings, although their significance is currently underestimated. This raises important questions about the role of small models in the era of LLMs, a topic that has received limited attention in prior surveys. In this work, we systematically examine the relationship between LLMs and SMs from two key perspectives: Collaboration and Competition (or Complementarity). We hope this survey provides valuable insights for practitioners, fostering a deeper understanding of the contribution of small models and promoting more efficient use of computational resources.
Sources
- Toxicity of the Commons: Curating Open-Source Pre-Training Data
- Small Language Models are the Future of Agentic AI
- ChatGPT may Pass the Bar Exam soon, but has a Long Way to Go for the LexGLUE benchmark
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Learning to Cascade: Confidence Calibration for Improving the Accuracy and Computational Cost of Cascade Inference Systems
- A Survey on LLM-as-a-Judge
- Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of Small Multilingual Language Models for Low-Resource Languages
- Tiny-Toxic-Detector: A compact transformer-based model for toxic content detection
- AutoScale: Scale-Aware Data Mixing for Pre-Training LLMs
- LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
- Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey
- Small Language Models: Survey, Measurements, and Insights
- HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models
- GPT-4 Technical Report
- FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
- Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLP
- Small Language Models (SLMs) Can Still Pack a Punch: A survey (updated 2026)
- A Survey of Small Language Models
- Cascade-Aware Training of Language Models
- A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering