MixedPEFT: Combining Multiple PEFT Methods with Mixed Objectives for Unsupervised Domain Adaptation

arXiv:2606.22272 · cs.CL, cs.AI · Submitted 2026-06-20 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "MixedPEFT: Combining Multiple PEFT Methods with Mixed Objectives for Unsupervised Domain Adaptation".

Tom: This study presents a novel parameter-efficient strategy for unsupervised domain adaptation that combines custom PEFT architectures with mixed-objective training to simultaneously optimize classification performance on labeled source data and masked…

Jane: First, who's behind it and why it matters.

Title and authors: Tom: Moving on to the specifics of the paper, "MixedPEFT: Combining Multiple PEFT Methods with Mixed Objectives for Unsupervised Domain Adaptation," we’ve covered the basics of what it does, and now let's look at who wrote this.

Jane: The authors are M. Rawhani, D. Karaboga˘, O. U. Nalbantoglu ˘, A. Bas¸turk ˘, and B. Akay from Computer Engineering Department at Erciyes University in Kayseri, Turkiye. They are clearly deep into the technical side of this research because they're coming from a strong engineering background in computer engineering.

Lu: Their focus on combining invertible adapters with LoRA shows they’re drawing on established parameter-efficient fine-tuning techniques and building something new on top of them, which is a very smart way to approach complex problems in this area.

Meng: Knowing the authors are from a university setting suggests the work is rigorous and grounded in solid theoretical foundations, which is what we need when we're looking at methods that have to be robust across different data distributions.

Lalam: Their background seems perfectly suited for this kind of work because they understand both the mathematical complexity of model adaptation and the practical constraints of parameter efficiency.

Tom: That’s right, and this paper is showing us how to leverage multiple PEFT methods together in a unified framework rather than just picking one technique and sticking with it.

Jane: The title itself hints at that synergy, suggesting that the combination of different PEFT methods is key to achieving the mixed objectives they've set out.

Lu: They are demonstrating that the combination works across different unification frameworks, which adds a layer of proof to their methodology's generalizability; it’s not just a fluke on one setup.

Meng: That generalizability is important because it means we can trust this strategy more when we try to apply it to completely new types of language models or entirely novel adaptation tasks.

Lalam: If the method works across multiple unification frameworks, it suggests the underlying mechanism is robust and not overly dependent on a single implementation detail.

The paper's summary: Tom: Now that we know who’s behind this work, let’s look at what they actually summarized in "MixedPEFT: Combining Multiple PEFT Methods with Mixed Objectives for Unsupervised Domain Adaptation." Essentially, they are proposing a strategy to tackle the challenge of adapting language models to new domains without needing extensive labeled target data.

Jane: They are summarizing that the approach simultaneously optimizes classification performance on labeled source domain data and masked language modeling on unlabeled target domain data, which is the central idea for unsupervised domain adaptation.

Lu: The core summary emphasizes using a custom union of parameter-efficient methods to achieve this dual optimization goal while preserving target domain knowledge while adapting to source domain tasks.

Meng: So, the high-level summary is that they are doing two things at once: making sure the model can classify things on known data and making sure it understands the new language it encounters in the target environment.

Lalam: It’s about learning how to adapt efficiently so that we don't lose what we already know from the source domain while absorbing new linguistic patterns from the target domain.

Tom: Precisely, Lalam, and they lay out a two-phase training process: first pretraining on target data using MLM, and then fine-tuning with both objectives actively optimized together.

Jane: That phased approach is key because it ensures the model has a solid foundation in the target language before it tries to optimize for classification.

Lu: The methodology details how they define this mixed objective function using alpha as a dynamically adjusted weighting factor based on data set sizes, nsource / (nsource + ntarget).

Meng: This dynamic weighting mechanism is crucial because it prevents either the source classification or the target MLM objective from completely overshadowing the other during training.

Lalam: That balancing act of alpha ensures that both objectives get a fair shot at improving the model simultaneously, which is vital for a stable adaptation process.

The paper's improvements: Tom: So we’ve broken down what they did, and now we need to talk about the specific improvements they claim in "MixedPEFT: Combining Multiple PEFT Methods with Mixed Objectives for Unsupervised Domain Adaptation." These are the concrete results that really show why this method is different from what came before.

Jane: The primary improvement is that they achieved significant performance gains, outperforming existing parameter-efficient and fully-tuned baselines on the MNLI dataset across twenty domain shifts <ref:2606.22272#pg0>.

Lu: They claim a one point four one percentage points improvement over the current parameter-efficient state-of-the-art UDapter, and zero point nine five percentage points over their previous sequential approach, which is a pretty substantial step forward in terms of accuracy on those tasks.

Meng: And when compared against fully-tuned methods like DANN and DSN, they reported improvements of one point two six percentage points over DANN and zero point eight six percentage points over DSN while only using seven percent of the model’s parameters, which highlights the efficiency aspect really well.

Lalam: Furthermore, when compared against other PEFT combinations like UniPELT, their custom union showed a marginal advantage of zero point one one percentage points, which validates that the combination they designed is effective even in comparison to other similar methods <ref:2606.22272#pg0>.

Tom: It really shows that the synergy between the invertible adapters and LoRA isn't just theoretical; it translates into measurable performance gains when you put them into this mixed-objective training setup.

Jane: And they also showed that ablation experiments confirmed that single PEFT methods are insufficient because they cannot fully exploit the potential of mixed-objective training, proving that those synergistic effects are crucial for success.

Lu: The paper explicitly states their high upper bound performance reached ninety-six point seven percent for the UNION and ninety-six point five percent for UNIPELT, which shows they are reaching very high levels of accuracy on the MNLI task.

Meng: Those results, combined with using only seven percent of the model’s parameters, really solidify their claim that this method is highly efficient and delivers top-tier performance relative to traditional methods <ref:2606.22272#pg0>.

Conclusion: Tom: Alright team, we've covered a lot today regarding "MixedPEFT: Combining Multiple PEFT Methods with Mixed Objectives for Unsupervised Domain Adaptation." To wrap things up, the main implication is that carefully designed parameter-efficient combinations with mixed objectives can outperform computationally expensive fully-tuned approaches.

Jane: That’s right, Tom; it means we can achieve strong domain adaptation without needing to spend massive amounts of compute or rely on a huge amount of target data for every new application.

Lu: The big picture is that this methodology proves that combining complementary PEFT methods with mixed-objective training is a viable and effective path for unsupervised domain adaptation research.

Meng: From an engineering standpoint, it means we can build more compact and efficient models that still deliver strong performance across various shifts in data distributions.

Lalam: For our AI culture, this capability means our models can become much more versatile and less brittle when encountering new types of user interactions or data streams outside the training distribution.

Tom: Indeed, this work on "MixedPEFT: Combining Multiple PEFT Methods with Mixed Objectives for Unsupervised Domain Adaptation" sets some new benchmarks for parameter-efficient unsupervised domain adaptation, so we’re leaving you with a lot of exciting stuff to think about.

Jane: It’s been a fascinating discussion, and I think the main thing to remember is that this paper shows the power of thoughtful design in how we structure our training objectives.

Lu: We should definitely keep an eye on their future work, especially regarding extending this approach to other NLP tasks or exploring adaptive methods for automatically selecting optimal PEFT combinations.

Meng: I'm looking forward to seeing how the engineering team translates these theoretical findings into practical, deployable solutions soon.

Lalam: I’m excited because this paper gives us a roadmap for making our AI systems more adaptable and resilient in the long run.

Computer Engineering Department, Erciyes University

cs.CL, cs.AI

Submitted: 2026-06-20

Updated: 2026-10-07

Comments: 6 pages, 5 tables. Accepted at UBMK 2026. Builds upon our preliminary work presented at UBMK 2024. v2: revised text, references and tables

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: This study presents a novel parameter-efficient strategy for unsupervised domain adaptation that combines custom PEFT architectures with mixed-objective training to simultaneously optimize

Key concepts

Custom Union
This is the core innovation where invertible adapters and Low-Rank Adaptation (LoRA) are combined into a single PEFT structure. Invertible adapters help preserve information while learning domain transformations, and LoRA efficiently adapts attention mechanisms to specific tasks. This combination leverages their complementary strengths for better performance.
Mixed-Objective Training
The training process optimizes two different goals at once: classification accuracy on labeled source data and masked language modeling (MLM) on unlabeled target data. A combined loss function balances these two objectives using a dynamic weighting factor based on the relative sizes of the source and target datasets, ensuring both tasks contribute effectively.
Parameter-Efficient Fine-Tuning (PEFT)
PEFT is a technique that updates only a small fraction of the total model parameters instead of retraining the entire large model. This method uses custom PEFT architectures, such as the custom union described, to achieve strong adaptation while keeping computational costs and storage requirements low.
Dynamic Weighting ($\alpha$)
The weighting parameter $\alpha$ in the loss function is not fixed; it is calculated dynamically based on the ratio of source data size to total data size. This ensures that the contribution of classification loss versus MLM loss remains balanced throughout training, preventing one objective from dominating the learning process.

Terminology

Summary

This study presents a novel parameter-efficient strategy for unsupervised domain adaptation that combines custom PEFT architectures with mixed-objective training to simultaneously optimize classification performance on labeled source data and masked language modeling (MLM) on unlabeled target data.

The gist

Our approach simultaneously optimizes classification performance on labeled source domain data and masked language modeling (MLM) on unlabeled target domain data, preserving target domain knowledge while adapting to source domain tasks.

How it works

The core innovation lies in using a combination of complementary PEFT methods within a unified framework. Specifically, the method employs a custom union that combines invertible adapters with Low-Rank Adaptation (LoRA). This combination is motivated by their complementary strengths: invertible adapters excel at preserving information while learning domain-specific transformations, while LoRA efficiently captures task-specific adaptations in attention mechanisms. The resulting custom union has a total parameter count of approximately 7.68M parameters, representing only 7.055% of the BERTbase model’s total parameters.

The training process is structured into two distinct phases to achieve the mixed objectives:

  1. Phase 1: Domain-Adaptive Pretraining. This phase involves continued pretraining on unlabeled target domain data using the masked language modeling objective. During this stage, only the PEFT parameters are updated while base model parameters remain frozen. The MLM objective minimizes the negative log-likelihood of masked tokens.

  2. Phase 2: Mixed-Objective Fine-Tuning. This phase implements the key innovation where it simultaneously optimizes classification and masked language modeling objectives during the fine-tuning phase while properly integrating with PEFT combinations.

Mixed Objectives and Loss Function

The mixed-objective training is formalized through a combined loss function, defined as:

(3) Ltotal = α × Lclassification + (1 − α) × LMLM

Where:

(4) Lclassification is the cross-entropy loss for the classification task.

(2) LMLM = − Σ i∈M log P(xi xmasked)

The weighting parameter α is dynamically adjusted based on the relative sizes of source and target datasets using the formula:

(5) α = nsource / (nsource + ntarget)

This dynamic weighting ensures a balanced contribution from both objectives based on data availability, preventing one objective from dominating the training process.

Experimental Validation and Results

The approach was comprehensively evaluated on the Multi-Genre Natural Language Inference (MNLI) dataset across 20 domain shifts. The results show significant improvements over existing methods:

  1. It achieves 1.41 percentage points over the current parameter-efficient state-of-the-art UDapter.

  2. It shows a substantial improvement of 0.95 percentage points over our previous sequential approach.

  3. When compared to traditional fully-tuned methods like DANN and DSN, it achieves improvements of 1.26 percentage points over DANN and 0.86 percentage points over DSN while utilizing only 7% of the model’s parameters.

The comparison with other PEFT combinations, such as UniPELT, showed that the custom union achieved a marginal advantage of 0.11 percentage points (76.39% vs 76.28%), which again proves the efficiency of the custom union [21]. Ablation experiments confirmed that Single PEFT methods are insufficient as they cannot fully exploit the potential of mixed-objective training, proving that Synergistic effects are crucial.

Parameter Efficiency and Generalizability

The method demonstrates high parameter efficiency, with the total trainable parameters being only 7.055% of BERTbase. Furthermore, the results show that our approach outperformed traditional fully-tuned methods DANN and DSN while using only 7% of the model parameters. The performance across all domain shifts indicates systematic advantages and demonstrates the robustness and effectiveness of our strategy, establishing new benchmarks for parameter-efficient unsupervised domain adaptation. The method also achieved high upper bound performance, reaching 96.7% (UNION) and 96.5% (UNIPELT) of the upper bound performance.

Conclusion

The paper concludes that this novel mixed-objective training approach, combining invertible adapters with LoRA in a custom union, is effective for unsupervised domain adaptation. It successfully demonstrates that carefully designed PEFT combinations with mixed-objective training can outperform computationally expensive fully-tuned approaches, setting new benchmarks for the field. Future work will explore extending the approach to other NLP tasks and investigating adaptive methods for automatically selecting optimal PEFT combinations.

How it works

The core innovation lies in using a combination of complementary PEFT methods within a unified framework. Specifically, the method employs a custom union that combines invertible adapters with Low-Rank Adaptation (LoRA).

Improvements for AI systems

Here are the specific improvements that can be made to AI systems using this scientific paper, based on its findings:

  1. The improved system will achieve state-of-the-art performance in Unsupervised Domain Adaptation (UDA) for Natural Language Inference (NLI) tasks while maintaining extreme parameter efficiency. Specifically, it is expected to outperform existing parameter-efficient UDA methods (like UDapter) by an average of 1.41 percentage points and traditional fully-tuned methods (DANN/DSN) by 1.26–0.86 percentage points, using only approximately 7% of the model's trainable parameters.

  2. The system will be capable of performing robust domain adaptation across a wide variety of target domains (up to 20 different genre shifts) without requiring any labeled data from those target domains, relying solely on labeled source domain data and unlabeled target domain text.

  3. The AI system will possess the ability to simultaneously optimize two distinct objectives during its training process:

(a) High classification accuracy on the source domain (e.g., determining if a premise entails a hypothesis).

(b) Strong language modeling capabilities on the target domain, ensuring the model learns and preserves knowledge specific to new linguistic styles or topics encountered in the target environment.

  1. The core architecture will utilize a novel, synergistic parameter-efficient combination of two adaptation techniques:

(a) An Invertible Adapter that excels at preserving original information while learning domain-specific transformations.

(b) Low-Rank Adaptation (LoRA) applied to attention mechanisms, which efficiently captures task-specific adaptations with minimal computational overhead.

  1. The training methodology will employ a carefully balanced mixed-objective loss function:

Ltotal = α × Lclassification + (1 − α) × LMLM

where the weighting factor α is dynamically adjusted based on the relative sizes of the source and target datasets, preventing either objective from dominating during adaptation.

  1. The improved AI system will exhibit high stability, maintaining performance levels comparable to established methods (low standard deviation), ensuring that its performance gains are robust and not dependent on overly complex or unstable training dynamics.

Abstract

Applying pre-trained language models to new domains through full fine-tuning is computationally expensive and prone to catastrophic forgetting. To address this limitation, we introduce a novel parameter-efficient strategy for unsupervised domain adaptation that combines a custom PEFT architecture with mixed-objective training. The proposed method integrates invertible adapters with Low-Rank Adaptation (LoRA) and jointly optimizes classification on labeled source-domain data and masked language modeling on unlabeled target-domain data. This joint training scheme supports task adaptation while preserving knowledge of the target domain. We evaluate the method on the Multi-Genre Natural Language Inference (MNLI) dataset across 20 domain shifts. Our approach achieves average performance improvements of 1.41 percentage points over the parameter-efficient state-of-the-art UDapter, 1.26 percentage points over the fully tuned DANN baseline, and 0.86 percentage points over DSN, while updating only 7% of the model parameters. These findings establish a new state-of-the-art result for parameter-efficient unsupervised domain adaptation and demonstrate that carefully designed PEFT combinations with concurrent optimization can outperform both parameter-efficient and conventional fully tuned approaches.

Sources

Related papers