PGSB: Pretrained-Guided Shared Basis for LoRA Model Merging

summary

The gist

The gist The proposed Pretrained-Guided Shared Basis (PGSB) is a two-stage framework for LoRA model merging that first identifies shared directions among task-specific LoRA updates and then

In short

PGSB is a two-stage framework for merging LoRA model updates by first finding shared directions across task-specific updates and then refining these directions using the pretrained weights' geometry. This process creates a compatible basis that allows for better merging, leading to state-of-the-art performance on vision and language tasks.

Key concepts

Task-shared Subspace Extraction (TSE): This stage identifies common directions among different task-specific LoRA updates. It achieves this by concatenating all updates and using Singular Value Decomposition (SVD) on the resulting matrices. This extracts a unified basis that captures the shared, task-relevant directions present across multiple tasks.
Pretrained-coupled Coordinate Refinement (PCR): This stage refines the initial shared basis by incorporating the geometric structure of the pretrained weights. It measures how much the pretrained weight couples with these task-shared directions and then uses SVD on this coupling matrix to concentrate dominant interactions onto diagonal coordinates, resulting in a more compatible coordinate system.
Pretrained-compatible Basis: This is the final output of PGSB, consisting of refined bases QL and QR. This basis preserves the original shared task subspaces while providing a coordinate system specifically tailored by the geometric structure of the pretrained model. It ensures that subsequent merging operations are performed in a space that minimizes conflicts with the original weight space.
Merging and Reconstruction: Once the refined bases are obtained, all task-specific LoRA updates are mapped into this new coordinate space. A merge rule is then applied in this simplified coordinate system to get a merged update, which is finally mapped back to the original parameter space using the refined basis for reconstruction.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 20 episodes
Federated Learning — explained in 17 episodes
Retrieval-Augmented Generation (RAG) — explained in 17 episodes
Quantum Fisher Information (QFI) — explained in 16 episodes
Flow Matching — explained in 15 episodes

This episode discusses

The paper

PGSB: Pretrained-Guided Shared Basis for LoRA Model Merging · Read on arXiv

Muqing Liu, Chongjie Si, Zhuoya Liu, Yuheng Jia

Chien-Shiung Wu College, Southeast University · MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University

Large pretrained models have achieved remarkable success across diverse domains, yet adapting them to multiple downstream tasks remains challenging: full fine-tuning is costly, while maintaining separate task-specific models is storage-inefficient and fails to produce a unified multi-task model. Low-Rank Adaptation (LoRA) offers an efficient alternative for Model Merging by representing each task adaptation as a low-rank update over frozen pretrained weights. However, existing LoRA merging methods often treat task-specific updates as directly comparable objects and merge them in their original low-rank parameter spaces. This overlooks two key factors: which update directions provide reliable shared support across tasks, and how these directions should be re-parameterized with respect to the pretrained weights before merging. To address this limitation, we propose Pretrained-Guided Shared Basis (PGSB), a two-stage framework for LoRA model merging. PGSB first identifies shared directions among task-specific LoRA updates, and then re-parameterizes these directions using their interaction with the pretrained weights, producing a more suitable basis for merging. Extensive experiments demonstrate that PGSB consistently outperforms state-of-the-art baselines across multiple benchmarks. Codes are available at https://github.com/Kitschnada/Pretrained-Basis.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "PGSB: Pretrained-Guided Shared Basis for LoRA Model Merging".

Jane: The gist The proposed Pretrained-Guided Shared Basis (PGSB) is a two-stage framework for LoRA model merging that first identifies shared directions among task-specific LoRA updates and then re-parameterizes these directions…

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, to summarize what the Pretrained-Guided Shared Basis paper claims, it’s a two-stage process. First, they do something called Task-shared Subspace Extraction or TSE to identify those shared directions among all the task updates by concatenating them and using singular value decomposition.

Jane: That sounds like they are trying to find a common subspace where all the task updates live together, which is smart because it groups similar tasks together. How does that first step work technically?

Tom: They concatenate all those individual task-specific LoRA updates and then apply singular value decomposition to extract a shared subspace, which gives them this unified multi-task basis that captures the common directions relevant to different tasks.

Lu: And to get the row space structure, they actually apply SVD again but on the concatenated transposed updates. That’s a clever trick because it lets them get both support matrices from the same singular value decomposition construction.

Meng: Okay, so they find this shared subspace first, and then in the second stage, Pretrained-coupled Coordinate Refinement or PCR, they refine that basis using the geometric structure of the pretrained weights to make sure it doesn't conflict with how those original weights are already set up.

Jane: That refinement step sounds crucial. So why is coupling with the pretrained weight geometry so necessary for this merging process? What happens if you just stick to a mathematically derived shared subspace without looking at the original model structure?

Lalam: The coupling quantifies it by taking T equal to sigma LG sigma R, which represents the pretrained-weight coupling after factoring in how important those task-shared directions actually are. They then compute the SVD of that T matrix to concentrate the dominant interactions on diagonal coordinates.

Tom: So they end up with these refined bases QL and QR that keep the original task-shared subspaces but give you a coordinate system that’s specifically tuned by the geometry of the pretrained weight structure.

Conclusion: Tom: So to wrap up on PGSB: this framework addresses those two big hurdles we talked about—finding shared directions and making them compatible with the pretrained weight geometry—by using TSE and then PCR.

Jane: It seems like they’ve shown that you need both the shared support construction and that refinement step, because just having a shared subspace isn't enough; it has to fit into the model structure itself.

Lu: The authors show that this two-stage approach consistently outperforms strong baselines on vision and language benchmarks, and they did an ablation study confirming that PCR gives them additional gains beyond just constructing the shared support.

Meng: From a deployment standpoint, the efficiency is also important. They’ve managed to avoid SVD on massive concatenated update matrices by using compact low-rank factors in their implementation, which means the preprocessing complexity is O(n 2Tr + T 3r three) <ref:2610.14558#pg1>. That keeps things manageable for large models.

Lalam: And they show that this method doesn't rely on a fixed or uniform LoRA rank, meaning it can be effective across diverse rank configurations, which is important for real-world scenarios where you don't know the optimal rank beforehand.

Tom: So what does this mean for the future? It suggests that for effective LoRA merging, we need both finding those shared task directions and making sure those directions align with how the pretrained model was originally trained.

Jane: And it points toward a future where model merging isn't just about picking the best update from a list, but about intelligently structuring those updates based on both their similarity to each other and their relationship to the core model.

More episodes

← Home