A Statistical Perspective on Knowledge Distillation: Foundations, Classical Methods, and Large Language Model Extensions
stat.ML, cs.LG
Submitted: 2026-09-27
Updated: 2026-09-27
Terminology
Sources
- Wearable Accelerometer Foundation Models for Health via Knowledge Distillation
- GPT-4 Technical Report
- Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
- Multi-Teacher Knowledge Distillation via Teacher-Informed Mixture Priors
- Keypoint-based Progressive Chain-of-Thought Distillation for LLMs
- MiniLLM: On-Policy Distillation of Large Language Models
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Distilling the Knowledge in a Neural Network
- TinyBERT: Distilling BERT for Natural Language Understanding
- Sequence-Level Knowledge Distillation
- Ensemble Distribution Distillation
- A Comprehensive Survey on Knowledge Distillation
- CourseGPT-zh: an Educational Large Language Model Based on Knowledge Distillation Incorporating Prompt Optimization
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- FitNets: Hints for Thin Deep Nets
- Patient Knowledge Distillation for BERT Model Compression
- Understanding and Improving Knowledge Distillation
- Assessing the Robustness of Bayesian Dark Knowledge to Posterior Uncertainty
- SCOTT: Self-Consistent Chain-of-Thought Distillation
- Dynamic Temperature Knowledge Distillation
Related papers
- Behavior of prediction performance metrics with rare events
- Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
- A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
- One Permutation Is All You Need: Fast, Deterministic Feature Importance and Model Stress-Testing
- Online Conformal Prediction for Non-Exchangeable Panel Data
- Deep Time-Series Forecasting in 10 Years: A Survey