Fine-Tuning of Transformer models with Frames
cs.AI
Submitted: 2026-08-26
Updated: 2026-08-26
Comments: 21 pages, 6 figures
Code: https://github.com/vsingh-group/FrameFT
License: http://creativecommons.org/licenses/by/4.0/
The gist: Parameter-Efficient Fine-Tuning (PEFT) strategies such as Low-Rank Adaptation (LoRA) are effective solutions for fine-tuning large-scale pre-trained models; however, their memory requirements scale
Terminology
Abstract
Parameter-Efficient Fine-Tuning (PEFT) strategies such as Low-Rank Adaptation (LoRA) are effective solutions for fine-tuning large-scale pre-trained models; however, their memory requirements scale with the size of the model, O(dr), where d is the model's hidden dimension and r is the rank. Our proposal, FrameFT, models the parameter update ΔW with a sparse coefficient matrix in a Fusion Frame basis. Fusion Frames can be generated algorithmically and shared across model layers, enabling very efficient updates. Only the sparse coefficients of the basis expansion are stored/optimized, reducing the memory footprint. The sparse structure of the coefficient matrix in FrameFT and the sparsity in the Fusion Frames give large compute benefits, and our analysis provides formal convergence results. We evaluate the idea across a suite of supervised fine-tuning benchmarks, focusing on language tasks, but also report application to vision models. Our experiments show that FrameFT achieves performance on par with/exceeding state-of-the-art PEFT techniques, but needs far fewer trainable parameters.
Sources
- Frame Quantization of Neural Networks
- LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters
- QLoRA: Efficient Finetuning of Quantized LLMs
- SingLoRA: Low Rank Adaptation Using a Single Matrix
- A Survey of Circuit Foundation Model: Foundation AI Models for VLSI Circuit Design and EDA
- Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
- RoBERTa: A Robustly Optimized BERT Pretraining Approach
- Choice of PEFT Technique in Continual Learning: Prompt Tuning is Not All You Need
- Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMs
- Learning How to Ask: Querying LMs with Mixtures of Soft Prompts
- SVFit: Parameter-Efficient Fine-Tuning of Large Pre-Trained Models Using Singular Values
- AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
- On the convergence properties of a $K$-step averaging stochastic gradient descent algorithm for nonconvex optimization
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection