MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora

MixLoRA-DSI is presented as a parameter-efficient fine-tuning (PEFT)-based dynamically expandable framework for rehearsal-free GR over dynamic corpora, designed to address the challenges of continual learning in generative retrieval (GR) systems that handle constantly changing document sets.

The core problem addressed by the paper is that continually updating model-based indexes in generative retrieval with new documents remains challenging, as full retraining is computationally expensive and impractical under resource constraints. Furthermore, traditional rehearsal-based methods are deemed impractical for real-world applications due to privacy concerns and the fact that dynamic corpora often require a task-agnostic, rehearsal-free solution.

MixLoRA-DSI integrates several novel components to achieve scalability and stability:

1. Mixture of LoRA Experts (MixLoRA):

The framework builds upon the Mixture of Experts (MoE) concept, where each expert is implemented as a low-rank adapter (LoRA). The model utilizes the T5 decoder blocks, replacing the original Feed-Forward Network (FFN) with MixLoRAs. The output of this layer is a weighted sum determined by a top- k selection:

MoE(x) = sum i=1 N top k(p i(x))E i(x)

2. Dynamic Expansion Strategy (OOD-driven):

To prevent linear parameter growth that occurs when naively adding new experts for each corpus update, MixLoRA-DSI employs a layer-wise OOD-driven expansion strategy. This approach ensures that expansion occurs only when necessary. The decision to expand is governed by detecting Out-of-Distribution (OOD) signals in the latent space.

• The energy score E(x; R) is calculated for each token: E(x; R) = -1 over T sum i (R i, x / T).

• A token is flagged as OOD if its energy score exceeds the corresponding threshold tau C, i.

A query is considered OOD if it contains at least one OOD token. If the number of OOD queries for a layer exceeds a predefined threshold delta,we trigger expert expansion for that layer, enabling sublinear parameter growth while maintaining retrieval effectiveness.

3. Improved Router:

The standard top-k linear router is replaced with a cosine classifier. This improvement is designed to mitigate the issue of recency bias in MoE routing, where tokens are predominantly routed to newly added experts while ignoring older ones. The improved router is optimized using a novel auxiliary loss (L aux) and incorporates a masking strategy:

z'i[j] = z i[j] & if m i[j]=1-infinity & if m i[j]=0

This ensures the softmax computation is restricted to the relevant segment of the extended vocabulary, preventing unnecessary competition between logits.

4. RQ-based Docids and CL Strategies:

The framework utilizes Residual Quantization (RQ)-based docids, which are more scalable than traditional atomic docids. Document embeddings (ED) are approximated by training M codebooks, each containing K centroids. These learned centroids are concatenated to the DSI’s original vocabulary weights (W vocab):

W RQ = W vocab; C 1;; C M

To enhance continual learning (CL) performance, a slow-learner strategy is employed, which involves scaling down the gradient updates of the output vocabulary W RQ, while keeping the original portion W vocab frozen. Furthermore, updates are regularized using KL divergence.

The optimization objective for MixLoRA-DSI combines two loss terms:

Loss = 1 over Q Dt L(Q Dt; theta t, theta t-1) + alpha 1 1 over X (sum i=1 X F i P i)

Where L is the log-softmax cross-entropy loss (LCE), and the second term is the auxiliary load balancing loss (L aux).

Evaluation metrics include:

• R@10 and M@10 (Retrieval Performance)

• Average Performance (AP)

• Backward Transfer (BWT, or Forgetting)

• Forward Transfer (FWT, or Learning Performance)

Experiments conducted on NQ320k and MS MARCO Passage demonstrate that MixLoRA-DSI outperforms full-model update baselines while maintaining minimal parameter overhead.

Key Findings:

1. Performance: MixLoRA-DSI achieves strong retrieval performance, with the best results noted in Table 1 at 78.0/70.0 (R@10/M@10).

2. Efficiency: The dynamic expansion strategy allows the model to achieve sublinear parameter growth.

3. Stability-Plasticity Trade-off: The results indicate that MixLoRA-DSI maintains a superior balance between stability and plasticity, achieving high AP while suffering much less from forgetting compared to competing methods like CLEVER.

In conclusion, the authors assert that MixLoRA-DSI is the first dynamically expandable framework for rehearsal-free GR over dynamic corpora, offering a solution that is task-agnostic, rehearsal-free, and scalable.

Based on a rigorous analysis of the MixLoRA-DSI framework, here are the specific improvements it offers to current AI systems, followed by a detailed description of what an AI system utilizing this architecture can achieve.

MixLoRA-DSI introduces five critical enhancements that fundamentally improve the efficiency and scalability of Generative Retrieval (GR) systems:

A. Dynamic, Rehearsal-Free Adaptation:

• Improvement: Instead of forcing a complete model retraining or relying on expensive, full-model continuous learning (CL), MixLoRA-DSI allows the system to learn incrementally from new data streams without needing access to previous documents (rehearsal). This is achieved by freezing the core backbone and adapting only specific components.

• Mechanism: The system utilizes a freeze-and-expand strategy.

B. Sublinear Parameter Growth via OOD Detection:

• Improvement: Unlike naive methods that add a new expert for every new corpus (leading to linear parameter growth), MixLoRA-DSI only adds a new Low-Rank Adaptation (LoRA) expert when the system detects genuinely novel information in the incoming data.

• Mechanism: This is governed by layer-wise Out-of-Distribution (OOD) detection using energy scores (E(x; R)). If OOD signals exceed a predefined threshold (delta), a new LoRA expert is selectively introduced.

C. Mitigation of Recency Bias via Improved Routing:

• Improvement: The system replaces the standard top-k linear router with a top-k cosine classifier. This critical change prevents the recency bias (where newly added experts dominate and ignore previously learned knowledge), ensuring that older, well-established expertise is maintained and utilized efficiently.

• Differentiator: It uses a novel auxiliary loss (L aux) to encourage expert specialization while maintaining balanced token assignment across multiple experts.

D. Scalable Knowledge Encoding (RQ-based Docids):

• Improvement: It utilizes Residual Quantization (RQ)-based document identifiers. This allows the system to encode massive, complex documents into a compact, scalable representation (M times K centroids) that is significantly more robust and scalable than traditional atomic docid methods.

• Mechanism: The the system learns to map document embeddings (E d) onto these learned RQ codebooks.

E. Optimized Continual Learning (CL) Objective:

• Improvement: The training objective function (L aux + alpha 2 LKL) is designed to simultaneously maximize learning new knowledge (Forward Transfer, FWT) and minimizing the degradation of previously learned knowledge (Backward Transfer, BWT).

• Tuning: By tuning alpha 1 (router balance) and alpha 2 (knowledge preservation), the system achieves a superior stability-plasticity trade-off.

An AI system built upon the MixLoRA-DSI architecture possesses the following operational capabilities:

A. Continuous, Uninterrupted Knowledge Acquisition:

• The system can operate in a truly dynamic environment, indexing and integrating new documents into its knowledge base indefinitely without requiring massive retraining cycles or needing to store large historical datasets for rehearsal.

B. Extreme Scalability for Infinite Corporas:

• The AI system can handle corpora that grow to millions of documents (e.g., MS MARCO scale) while maintaining computational efficiency and preventing the exponential degradation of performance due to linear parameter bloat.

C. High Retrieval Precision with Knowledge Preservation:

• The system excels at retrieving relevant information from both the most recent data and the foundational, previously indexed knowledge, ensuring that knowledge gained in earlier stages is not forgotten (low BWT). This makes it highly reliable for long-term, real-world applications.

D. Resource Efficiency:

• By using PEFT (LoRA) and only expanding when necessary (OOD detection), the system requires significantly fewer trainable parameters compared to full fine-tuning methods, drastically reducing computational cost during both indexing and inference.

E. Targeted Adaptation:

• The system intelligently allocates its learning effort. It does not waste computational resources attempting to learn concepts already well-represented in the model; it focuses its limited parameter budget on capturing only the genuinely novel semantic information present in new documents.

Continually updating model-based indexes in generative retrieval with new documents remains challenging, as full retraining is computationally expensive and impractical under resource constraints. We propose MixLoRA-DSI, a novel framework that combines an expandable mixture of Low-Rank Adaptation experts with a layer-wise out-of-distribution (OOD)-driven expansion strategy. Instead of allocating new experts for each new corpus, our proposed expansion strategy enables sublinear parameter growth by selectively introducing new experts only when significant number of OOD documents are detected. Experiments on NQ320k and MS MARCO Passage demonstrate that MixLoRA-DSI outperforms full-model update baselines, with minimal parameter overhead and substantially lower training costs.