InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models".

Jane: The gist The current few fusion methods on PA phase, like WRPO,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: The paper’s title, "InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models," really tells you exactly what it does—it's about fusing models implicitly through preference optimization. They list Yanggan Gu, Yuanyi Wang, Zhaoyi Yan, Yiming Zhang, Qi Zhou, Fei Wu, and Hongxia Yang as the authors who developed this framework.

Jane: It’s interesting how they frame it as implicit fusion. This isn't about explicitly merging the model weights in a traditional sense; it’s about creating a fused source model to serve as the reference during preference optimization steps, which is what they focus on.

Lu: The implication here is that we can achieve this integration without having to do complicated vocabulary matching between models, which has been a real hurdle for prior work like WRPO six <ref:2505.13878#pg2>.

Meng: So, if they solve the vocabulary alignment problem by looking at sequence probabilities instead of just token outputs, that suggests a more robust way to combine models with different underlying architectures.

Lalam: It opens up a path where the pivot model can truly benefit from the diverse knowledge each source model brings, not just one narrow perspective.

The paper's summary: Tom: The core idea of this paper is that existing methods on preference alignment often simplify things by only looking at response outputs and throwing away the probability information from the source models. InfiFPO goes a step further by replacing the reference model in DPO with a fused source model that synthesizes multi-source probabilities at the sequence level, which maintains those crucial probability signals.

Jane: They are moving from just using response-level supervision to using sequence probabilities from all source models, including for both preferred and dispreferred responses. That makes the fusion process much more comprehensive.

Lu: The framework is built on a KL-constrained optimization problem where you keep the pivot model within a certain distance, an epsilon ball, of every single source model simultaneously across the dataset D <ref:2505.13878#pg3>.

Meng: So, they’re using this constraint to force the pivot model to stay close in behavior to all sources at once while maximizing preference rewards. That sounds like a very structured way to learn from multiple models at the same time.

Lalam: This means the pivot model learns not only what humans prefer but also it learns from the probabilistic tendencies of every source model in that constrained space, which is a significant improvement over just one reference point.

The paper's improvements: Tom: They introduce three main enhancements to make this approach more stable and effective. First, they have length normalization to reduce bias caused by different token sequence lengths across the models. Second, there's probability clipping to limit the influence of underperforming source models by suppressing noisy gradients.

Jane: And third, they have max-margin fusion, which adaptively prioritizes the source models that provide the most distinctive and informative differences when comparing them to the pivot model. That sounds like a smart way to handle sources that might be less reliable.

Lu: These specific enhancements help improve robustness, especially when you're dealing with heterogeneous models where some might be performing better on certain tasks than others <ref:2505.13878#pg2>.

Meng: From an engineering standpoint, the probability clipping and length normalization are practical because they directly address common noise issues we see in training across different model sizes and vocabularies.

Lalam: Max-margin fusion is particularly interesting because it doesn't treat all sources equally; it actively seeks out the most informative models to guide the pivot model's learning.

Conclusion: Tom: So, to wrap up, InfiFPO enables model fusion during preference alignment by replacing the reference model in DPO with this fused sequence-level distribution over multiple source models, effectively bypassing token-level vocabulary issues while keeping the probability signals intact. They show it performs well on benchmarks like Phi-four improving its average performance from seventy-nine point nine five to eighty-three point three three on eleven tasks.

Jane: The combination of length normalization and probability clipping gave them the strongest performance boost across all those benchmarks, with gains as high as plus five point six in coding tasks and plus three point one in math ones.

Lu: It’s also worth mentioning that their max-margin fusion strategy outperformed both average-based and confidence-based strategies, showing it prioritizes the most useful source models.

Meng: While the results are strong on those specific benchmarks, the paper points out that they only selected five mainstream open-source LLMs for their experiments, so we don't yet know if this scales to even larger or more diverse model sets.

Lalam: They also note that more rigorous theoretical analysis is still needed to fully understand the fusion mechanisms behind InfiFPO, which is something they are looking into next.

Tom: Exactly. So, "InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models" gives us a principled framework for integrating diverse LLMs into a single model during preference alignment by leveraging full sequence probabilities instead of just token-level outputs. We'll be talking about other papers next on how AI systems are detecting free-riders in federated learning.