Harmony in Diversity: Multi-domain Contrastive Policy Optimization for Large Reasoning Models

arXiv:2605.25443 · cs.CL · Submitted 2026-05-25 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Harmony in Diversity".

Jane: Multi-domain Contrastive Policy Optimization (MCPO) is a structured contrastive reinforcement learning framework proposed to enhance Large Reasoning Models by transforming cross-domain interactions from harmful competition into beneficial knowledge transfer.

Tom: First, who's behind it and why it matters.

Paper summary: Jane: So, wrapping up our discussion on "Harmony in Diversity: Multi-domain Contrastive Policy Optimization for Large Reasoning Models," the authors are essentially arguing that their method offers a different pathway for scaling LRMs across diverse tasks. They show that by focusing on structural relationships, we can move past just mitigating interference and start actively building knowledge bridges between domains.

Tom: I agree. The authors of this paper emphasize that the title, "Harmony in Diversity," reflects their goal: to achieve a state where models don't just survive different environments but actually learn from them constructively. It suggests that diversity shouldn't be a source of friction for learning, but rather a rich resource for transfer.

Lu: The implication I see is that this moves us toward building truly versatile reasoning systems, not just specialized ones. If we can reliably consolidate in-domain knowledge while simultaneously pulling in useful insights from other domains, the potential for complex problem-solving across vastly different fields becomes much more accessible.

Meng: Practically speaking, this means that models trained with this approach might require less domain-specific fine-tuning if the core reasoning structure is robust enough to handle cross-domain transfer automatically. It addresses the challenge of making a single large model perform well everywhere without needing dozens of highly specialized versions.

Lalam: For me, this advance in knowledge consolidation could lead to a much more coherent and robust internal representation within the AI itself. If the model can consolidate correct knowledge from one area while retaining transferable structures from another, it builds a far more reliable foundation for complex reasoning tasks overall.

Tom: It’s clear that they are pushing toward a system where knowledge isn't siloed; it's interconnected in a way that benefits all of them. This paper suggests that the path forward isn't just about better sampling controls, but about designing learning objectives that reward genuine knowledge association across boundaries.

Jane: Exactly. The paper points out how their method addresses the limitations of prior work by focusing on positive knowledge associations rather than just conflict reduction. It’s a refinement of the RL training objective specifically designed for multi-domain contexts.

Lu: I think the real excitement here is seeing how these structural relationships, which we analyze through those contrastive objectives, can be leveraged to create novel reasoning pathways that we haven't even conceived of yet. It opens up a whole new way to think about how large models acquire and structure their understanding.

Meng: From an engineering standpoint, the authors are tackling the complexity of heterogeneous prompt groups updating a single policy under different dynamics, which is exactly where practical deployment gets messy. Their focus on explicit knowledge sharing seems like a very necessary step to make that update process more stable and beneficial.

Lalam: I think the most impactful vision here is realizing an AI culture where different specialized knowledge areas aren't treated as separate silos, but as interconnected parts of one unified reasoning entity. That level of internal coherence would fundamentally alter how we build and trust these large models.

Tom: Fantastic points, everyone. So, to summarize this paper on "Harmony in Diversity: Multi-domain Contrastive Policy Optimization for Large Reasoning Models," it proposes a structured contrastive approach that analyzes rollout relationships to explicitly promote cross-domain knowledge sharing and in-domain consolidation. It argues this is a more effective way to handle multi-domain RL than focusing solely on interference control, leading toward models that can better integrate diverse knowledge resources.

Conclusion: Tom: So, we’ve been diving deep into this paper by exploring how they structure learning across different areas of knowledge for large models.

Jane: It really is fascinating how they take the idea of different domains and turn that potential friction into a structured way for the model to learn from one another.

Lu: I find the way they use those contrastive objectives, specifically separating cross-domain sharing from in-domain consolidation, incredibly elegant when you think about the architecture of knowledge itself.

Meng: From an engineering standpoint, it’s interesting that they're using structural relationships among rollouts as a basis for their contrastive pool; that sounds like a very deliberate way to guide the policy updates.

Lalam: I see this as a fundamental step toward creating an AI culture where different specialized areas don't just exist side-by-side, but actively reinforce each other’s learning pathways.

Tom: Exactly! When we look at the title, "Harmony in Diversity," it really captures that core concept of finding balance within complexity.

Jane: And the authors are doing a fantastic job of showing us that this isn't just about making things less messy; it’s about building a more cohesive intelligence.

Lu: The implication here is huge for how we train models; instead of treating each domain as a separate silo, we’re designing the training process to encourage genuine, structured knowledge transfer between those silos.

Meng: I worry about the practical implementation detail of maintaining that balance; getting the weights for cross-domain versus in-domain consolidation just right under real-world constraints sounds tricky.

Lalam: But that’s where the vision becomes clear; if we can build this kind of internal coherence, it could lead to AI systems that exhibit a far more nuanced and adaptable form of reasoning than we see now.

Tom: That adaptability is what really excites me about this research; it suggests a path toward models that are less brittle when encountering novel or mixed-domain problems.

Jane: It gives us hope that the next generation of large models won't just be bigger, but fundamentally smarter in how they connect disparate pieces of information.

Lu: And thinking about the future work, I wonder if we could extend this structural relationship analysis to other forms of knowledge representation beyond just binary rewards.

Meng: I’d be curious to see if this framework can scale efficiently when we move from a few domains to hundreds, because that’s where the computational demands really start to climb.

Lalam: Ultimately, my biggest vision is an AI culture where different specialized knowledge areas aren't treated as separate silos, but actively reinforce each other’s learning pathways.

Tom: That focus on reinforcement across boundaries is what makes this paper so compelling; it shifts the goal from simply mastering one thing to mastering the connections between things.

Zongji Yu, Wenshui Luo, Yiliu Sun, Hao Fang, Runmin Cong, Chaochao Lu, †Chen Gong

Shanghai Jiao Tong University · Shanghai AI Laboratory

cs.CL

Submitted: 2026-05-25

Updated: 2026-09-28

Code: https://github.com/Maricalce/MCPO

Importance score: 83/100

The gist: Multi-domain Contrastive Policy Optimization (MCPO) is a structured contrastive reinforcement learning framework proposed to enhance Large Reasoning Models by transforming cross-domain interactions

Key concepts

MCPO
MCPO is a structured contrastive reinforcement learning framework designed to enhance large reasoning models. It explicitly models relationships between different training rollouts to promote two goals: sharing useful knowledge between domains and consolidating accurate information within the same domain.
GRPO Objective
The Group Relative Policy Optimization (GRPO) objective is the base objective used in MCPO. It helps optimize the policy by focusing on how actions within a group of rollouts relate to each other, which forms the foundation for the overall learning process.
Cross-domain Knowledge Sharing ($\mathcal{L}_{cross}$)
This objective encourages models to learn from other domains. Correct rollouts from different domains are treated as positive examples, and their similarity is used to weight the alignment of representations, allowing knowledge to be transferred between distinct areas.
In-domain Knowledge Consolidation ($\mathcal{L}_{in}$)
This objective focuses on strengthening knowledge within a single domain. Correct rollouts from the same prompt or other prompts in that same domain are treated as positive examples, pushing their representations closer together to build a robust, consolidated understanding of that specific area.

Terminology

Summary

Multi-domain Contrastive Policy Optimization (MCPO) is a structured contrastive reinforcement learning framework proposed to enhance Large Reasoning Models by transforming cross-domain interactions from harmful competition into beneficial knowledge transfer. This method addresses the limitation of existing multi-domain RL methods, which often focus solely on alleviating interference through sampling control or reward normalization, by introducing a mechanism that explicitly models structural relationships among rollouts to promote both cross-domain knowledge sharing and in-domain knowledge consolidation.

The gist

MCPO analyzes the structural relationships among rollouts and promotes cross-domain knowledge sharing and in-domain knowledge consolidation in a contrastive manner.

How it works

MCPO is a structured contrastive reinforcement learning framework that combines the within-prompt Group Relative Policy Optimization (GRPO) objective with two explicit contrastive objectives: cross-domain knowledge sharing and in-domain knowledge consolidation. The overall optimization objective is formulated as:

(12) Ltotal(theta, phi) = LGRPO(theta) + lambdaclLMCPO(theta, phi).

The framework operates by constructing a structured contrastive pool from the same mixed-domain rollout batch and mapping rollouts into a compact representation space through a lightweight contrastive head. This is inspired by the InfoNCE loss. The process involves:

  1. Identifying transferable reasoning trajectories from other domains as positive examples, while treating incorrect rollouts as negative ones for cross-domain knowledge sharing.

  2. Aligning intra-domain correct rollouts to build a consolidated representation space for in-domain knowledge consolidation.

Rollout-Level Representations

MCPO realizes structural rollout relationships by first constructing a representation space using the binary rewards from the verifier, where correct rollouts are positive examples and incorrect ones are negative ones. For each rollout, its hidden representation is extracted and projected into a low-dimensional embedding space:

(4) vp,y = norm(Pphi(hp,y)).

The mixed-domain rollout batch is then flattened into a set of rollouts denoted as I = (pi, yi, di, zi, vi), where the domain index is di and the binary reward is zi.

Prompt Prototypes and Contrastive Objectives

To facilitate cross-domain knowledge sharing and in-domain consolidation, MCPO defines two sets of positive examples for each correct anchor (where zi = 1):

(6) The cross-domain sharing set contains correct rollouts from other domains, with their contribution controlled by omegacij.

(7) The in-domain consolidation set includes correct rollouts from the same prompt and correct rollouts from other prompts in the same domain.

The cross-domain knowledge sharing objective (Lcross) uses prototype similarity to weight positive pairs, controlled by compatibility weights omegacij:

(9) Lcross = −1/Ncross ∑i zi=1 ∑j∈Pcrossi omegacij log exp(v⊤i vⱼ/τ) exp(v⊤i vⱼ/τ) + ∑k̸=i,j exp(v⊤i vk/τ).

The in-domain knowledge consolidation objective (Lin) counts positive pairs uniformly within the same domain:

(10) Lin = −1/Nin ∑i zi=1 ∑j∈Pini log exp(v⊤i vⱼ/τ) exp(v⊤i vⱼ/τ) + ∑k̸=i,j exp(v⊤i vk/τ).

The final contrastive objective is a weighted sum:

(11) LMCPO = lambdacrossLcross + lambdainLin.

Empirical Results and Analysis

Empirical results demonstrate that MCPO improves the reasoning capabilities of LRMs across multiple domains, with MCPO-DAPO showing significant gains over single-domain training baselines. The analysis reveals that:

  1. The contrastive pool becomes more informative as training proceeds, with the average number of correct positive examples growing and the mean cross-domain positive weight increasing from about 0.75 to 0.88 (Figure 2).

  2. Transfer-aware dynamics show that transferable reasoning trajectories are increasingly aligned as positive examples, while incorrect rollouts are pushed apart as negative ones in the contrastive space (Figure 3).

Improvements for AI systems

As a fastidious researcher, I have analyzed the core contribution of Harmony in Diversity: Multi-domain Contrastive Policy Optimization for Large Reasoning Models (MCPO). The proposed method fundamentally shifts multi-domain RL from simple conflict avoidance to structured knowledge transfer and consolidation.

Here are the specific improvements and capabilities this system enables:

  1. The ability to maintain a unified, harmonious representation space across diverse domains (Math, Code, Science, ToolUse, Safety) within a single Large Reasoning Model (LRM).

  2. The capability to perform effective cross-domain knowledge sharing by identifying and leveraging transferable reasoning trajectories from one domain as positive examples for another domain's problem.

  3. The capacity to consolidate in-domain knowledge by aligning different correct reasoning paths from the same domain into a robust, reusable structure, preventing fragmentation during joint training.

  4. The ability to distinguish between useful cross-domain interactions (compatible trajectories) and harmful interference or irrelevant structural competition (incorrect rollouts).

  5. Specific AI System Improvements:

  • An LRM trained via MCPO can solve complex, multi-faceted problems by drawing analogous reasoning structures from specialized domains. For example, it could apply the algebraic reasoning patterns learned from the Math domain to interpret and solve a complex physical problem in the Science domain (as demonstrated in Case 1).

  • The model will exhibit superior performance on benchmarks requiring cross-domain transfer, such as applying mathematical principles to scientific inquiry (e.g., solving problems requiring both calculus and physics concepts simultaneously), leading to higher scores than single-domain or standard multi-domain RL methods like DAPO or MGS.

  • The system will demonstrate better generalization by retaining domain-specific strengths while avoiding the fragmentation seen in methods that only focus on conflict suppression (like MGS). It will preserve reusable code structures from the Code domain while learning new scientific concepts, rather than letting the two knowledge bases become isolated clusters.

  • In safety alignment (Safety), MCPO allows for a more nuanced understanding of safe responses by contrastively comparing different reasoning paths across domains, potentially leading to a more robust and context-aware safety mechanism that is less prone to over- or under-filtering compared to methods like MT-GRPO which focus primarily on task balancing.

  1. Specific AI System Capabilities:

  • The LRM can generate solutions that are not only mathematically sound but also structurally consistent with the reasoning patterns of other domains, leading to more elegant and verifiable outputs (e.g., producing a physically correct work calculation instead of a mathematically derived but contextually incorrect one).

  • The model will exhibit enhanced reasoning coherence across its diverse knowledge base because the contrastive mechanism forces the representation space to be structured by structural relationships rather than just by raw reward maximization, resulting in reasoning chains that are more logically linked across domains.

Sources

Related papers