Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning

summary

The gist

The gist The proposed Low-rank Decomposition and Adaptation (LoDA) framework performs a task-driven decomposition to build general and truly task-specific LoRA subspaces by solving two energy-based

In short

The Low-rank Decomposition and Adaptation (LoDA) framework addresses limitations in continual learning by decomposing LoRA updates into general and task-specific subspaces. It uses energy-based objectives to build a general subspace for shared knowledge transfer and an isolated subspace for task-specific adaptation, improving knowledge sharing while minimizing interference from past tasks.

Key concepts

General Subspace (UG): This subspace captures directions salient across all tasks by maximizing the overall projection energy of both old and new tasks. It enables effective knowledge transfer between different learning stages by identifying common features that are relevant to multiple previous and current tasks.
Isolated Subspace (UI): This subspace identifies truly task-specific updates by maximizing the ratio of new-to-old relative energy. This isolates directions that are highly effective for the new task while having minimal impact on previously learned tasks, ensuring strong adaptation without interference.
Projection Energy: Projection energy quantifies how much a feature vector lies within a specific subspace defined by a set of vectors. It is used to motivate the design of the decomposition, guiding the selection of subspaces that capture either general knowledge (high energy across tasks) or task-specific novelty (high relative energy).
Gradient-Aligned Optimization (GAO): GAO is an optimization approach used to train the up-projections in LoDA. It encourages gradient consistency across label-disjoint subsets, which promotes shared directions between different tasks. This helps enlarge the set of safe update directions for the current task, making it more robust.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 18 episodes
Federated Learning — explained in 17 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes
Quantum Fisher Information (QFI) — explained in 15 episodes

This episode discusses

The paper

Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning · Read on arXiv

Lingfeng He, De ChengB, Huaijie Wang, Xi Yang, Nannan Wang, Xinbo Gao

Xidian University

Continual Learning (CL) requires models to sequentially adapt to new tasks without forgetting old knowledge. Recently, Low-Rank Adaptation (LoRA), a representative Parameter-Efficient Fine-Tuning (PEFT) method, has gained increasing attention in CL. Several LoRA-based CL methods reduce interference across tasks by separating their update spaces, typically building the new space from the estimated null space of past tasks. However, they (i) overlook task-shared directions, which suppresses knowledge transfer, and (ii) fail to capture truly effective task-specific directions since these ``null bases" of old tasks can remain nearly inactive for new task under correlated tasks. To address this, we study LoRA learning capability from a projection energy perspective, and propose Low-rank Decomposition and Adaptation (LoDA). It performs a task-driven decomposition to build general and truly task-specific LoRA subspaces by solving two energy-based objectives, decoupling directions for knowledge sharing and isolation. LoDA fixes LoRA down-projections on two subspaces and learns robust up-projections via a Gradient-Aligned Optimization (GAO) approach. After each task, before integrating the LoRA updates into the backbone, LoDA derives a closed-form recalibration for the general update, approximating a feature-level joint optimum along this task-shared direction. Experiments indicate that LoDA outperforms existing CL methods. Our code is available at https://github.com/HHHLF/LoDA ICML2026.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning".

Jane: The gist The proposed Low-rank Decomposition and Adaptation (LoDA) framework performs a task-driven decomposition to build general and truly task-specific LoRA subspaces by solving two energy-based objectives,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So wrapping up the "Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning" paper, the core idea is using two distinct subspaces, UG and UI, to manage how LoRA updates interact with old knowledge.

Jane: It’s about decoupling those directions so you get a general space for sharing what’s common and a specific space for learning what’s new without messing up the past. The authors claim this approach finds a better stability-plasticity trade-off than previous methods.

Lu: The implication is that we can design the LoRA subspaces based on task statistics—the projection energy—rather than just assuming a random split, which helps preserve transferable directions while still learning new things <ref:2603.00191#pg2>.

Meng: For deployment, this means models could adapt to new data more reliably because they are explicitly constrained from interfering with past tasks in certain ways. But the paper also notes a limitation, which is that it relies on estimating statistics from past tasks and then deriving the subspaces based on those estimates.

Lalam: Yes, one limitation mentioned is that the framework requires running a frozen backbone weight Wt-one on Dt to cache per-layer inputs and compute statistics S t before deriving UG and UI <ref:2603.00191#pg3>.

Tom: So, while it's theoretically sound with these two energy objectives, the practical cost is the need for that pre-processing step to get those statistics before you can even start learning for the new task.

Jane: That’s a fair point. The paper shows that LoDA outperforms existing continual learning methods on their experiments, but whether it’s efficient enough to run on a massive scale remains something we have to test further.

Conclusion: Jane: So, to wrap up this paper on Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning, they’ve essentially figured out how to build two separate learning spaces for a single model update.

Tom: Right, so it takes the old knowledge and the new task and creates a general space that lets them share stuff, but also a separate space that keeps things isolated for the new task.

Lu: Exactly. They solve two energy objectives to find those subspaces: one subspace for general directions, UG, which captures what’s common across all tasks.

Meng: And another subspace, UI, which they define as having the biggest ratio of new-to-old relative energy. That’s the part that should isolate the new task's learning.

Lalam: I see how it works conceptually; they use projection energy to gate which directions are learnable and which are fixed. It seems robust because it’s driven by data statistics, not just a random guess.

Jane: But what does this actually mean for someone just listening? It means we can design the model's update structure itself based on how the tasks overlap, instead of just hoping the standard LoRA setup works fine across different learning steps.

Tom: It’s about controlling that interference you talked about earlier. They use a Gradient Aligned Optimization technique to make sure those specific isolated updates are actually useful for the new task without hurting old ones.

Lu: The math shows that they can even derive a closed-form recalibration for the general update after each task, which helps approximate the best possible joint optimum.

Meng: For me, if we could build a system where we know exactly which directions are safe to change and which are strictly for new information, deployment would be much less risky at scale.

Lalam: It confirms that this decomposition is not just theoretical; the visualization of the loss landscapes shows those isolated directions actually lead to task-specific changes while keeping old tasks stable.

Jane: It’s a solid approach for making continual learning more principled by explicitly separating knowledge transfer from adaptation. But we need to see how this scales with a whole library of tasks.

Tom: Yeah, the authors are proving that by using energy-based objectives, you can get both general and task-specific directions simultaneously in LoRA.

Lu: The big implication is that the structure of the adaptation mechanism matters just as much as the data itself for continual learning systems.

Jane: Next up, we're going to look at how these subspace ideas compare to other ways people are tackling forgetting in AI models.

More episodes

← Home