Statistical Inference for Rank Allocation in Low-Rank Adaptation

arXiv:2607.20205 · stat.ML, cs.LG, math.ST · Submitted 2026-07-22 · Read on arXiv

Yihang Gao, Vincent Y. F. Tan

stat.ML, cs.LG, math.ST

Submitted: 2026-07-22

Comments: 77 pages

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Fine-Tuning — 60× in this paper · explained in 5 episodes
• Adaptation — 48× in this paper · explained in 1 episode
• Adam — 29× in this paper · explained in 2 episodes
• Verification — 21× in this paper · explained in 2 episodes
• LoRA Fine-Tuning — 19× in this paper · explained in 3 episodes
• Pruning — 19× in this paper · explained in 4 episodes
• Accuracy — 14× in this paper · explained in 1 episode
• Consistency — 9× in this paper · explained in 3 episodes
• Objective Function — 9× in this paper · explained in 2 episodes
• Parameter-Efficient Fine-Tuning — 7× in this paper · explained in 2 episodes
• Benchmark — 5× in this paper · explained in 2 episodes
• Generalization — 4× in this paper · explained in 13 episodes
• Low-Rank Adaptation (LoRA) — 4× in this paper · explained in 9 episodes
• Calibration — 3× in this paper · explained in 7 episodes
• Score Function — 3× in this paper · explained in 4 episodes
• Uncertainty Quantification — 3× in this paper · explained in 8 episodes
• Foundation Model — 2× in this paper · explained in 3 episodes
• Foundation Models — 2× in this paper · explained in 7 episodes
• Loss Function — 2× in this paper · explained in 2 episodes
• Mean-Field Dynamics — 2× in this paper · explained in 2 episodes
• Regularization — 2× in this paper · explained in 3 episodes
• Semantic Similarity — 2× in this paper · explained in 2 episodes
• Transfer Learning — 2× in this paper · explained in 10 episodes
• Transformer — 2× in this paper · explained in 2 episodes
• Ablation — 1× in this paper · explained in 1 episode
• Ablation study — 1× in this paper · explained in 1 episode
• Attention Mechanism — 1× in this paper · explained in 3 episodes
• Computational overhead — 1× in this paper · explained in 2 episodes
• Downstream Tasks — 1× in this paper · explained in 2 episodes
• F1 Score — 1× in this paper · explained in 3 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Stochastic Dynamics — 1× in this paper · explained in 2 episodes
• Unified Framework — 1× in this paper · explained in 5 episodes

Sources

• A Polyak-Ruppert Central Limit Theorem for SA-Adam with Momentum and Non-Convergent Adaptive Preconditioning
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Central limit theorem for the averaged Adam optimizer
• BaRA: Bayesian Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning
• Online Covariance Matrix Estimation in Sketched Newton Methods
• Derivative-Free Sequential Quadratic Programming for Equality-Constrained Stochastic Optimization
• Qwen2.5 Technical Report

Related papers

• Behavior of prediction performance metrics with rare events
• Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
• A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
• One Permutation Is All You Need: Fast, Deterministic Feature Importance and Model Stress-Testing
• Online Conformal Prediction for Non-Exchangeable Panel Data
• Deep Time-Series Forecasting in 10 Years: A Survey