UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hacking

summary

The gist

The gist The UMM-RM framework proposes an Upcycle-and-Merge Mixture-of-Experts Reward Model to mitigate reward hacking in Reinforcement Learning from Human Feedback by combining expert diversity with

In short

UMM-RM proposes an Upcycle-and-Merge Mixture-of-Experts Reward Model to stop reward hacking in RLHF. It copies layers from a pre-trained model to create shared and specialized experts, then merges them back into a single dense model using learnable weights. This method maintains expert diversity for robustness while keeping the efficiency of a standard reward model.

Key concepts

Reward Hacking: This occurs when an AI policy exploits biases or weak points in the reward model to maximize its score without actually improving alignment with human preferences. The model learns to generate outputs that trick the reward function, leading to poor final quality.
Mixture-of-Experts (MoE) Reward Model: This architecture uses multiple specialized neural networks (experts) to handle different types of preference signals. Some experts focus on general instructions, while others specialize in specific tasks or fine-grained preferences across various prompts, allowing for richer preference modeling.
Upcycle-and-Merge Strategy: The model first initializes experts by copying layers from a Stable Fine-Tuned (SFT) checkpoint. After training, these diverse experts are consolidated back into one dense reward model using learnable weights. This retains the benefits of having many specialized views while eliminating the computational cost of running multiple models.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 18 episodes
Federated Learning — explained in 17 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes
Quantum Fisher Information (QFI) — explained in 15 episodes

This episode discusses

The paper

UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hacking · Read on arXiv

Lingling Fu†, Yongfu Xue†

Guangxi University

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hacking".

Jane: The gist The UMM-RM framework proposes an Upcycle-and-Merge Mixture-of-Experts Reward Model to mitigate reward hacking in Reinforcement Learning from Human Feedback by combining expert diversity with parameter consolidation.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we’ve looked at the abstract and summary of this paper, "UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hacking." The main thrust here is about fixing reward hacking in RLHF.

Jane: They point out that dense reward models are prone to being exploited by policy models through biases or spurious correlations, which causes the model scores to inflate while its actual alignment with human preferences gets worse, particularly when there's a shift in data distribution.

Lu: To address this risk, UMM-RM proposes an Upcycle-and-Merge Mixture-of-Experts reward model. They start by upscaling the feedforward layers of a dense backbone into an MoE reward model that has shared experts and routed experts.

Meng: The key mechanism here is that the shared experts are always active to catch general preference signals, while the routed experts handle the fine-grained preferences across different instructions or tasks.

Tom: After training, they consolidate these experts back into a single dense reward model using learnable merging weights to compress the MoE structure.

Jane: This merging step is crucial because it reduces those high-variance reward responses from individual misaligned experts and helps weaken the model's reliance on those spurious features that cause the hacking.

Lu: They claim this design preserves the robustness and exploitation resistance that comes from having expert diversity while avoiding the inference overhead you usually see with MoE architectures or when you have to query an ensemble.

Tom: So, in short, they are proposing a way to get better reward supervision by combining expert diversity with parameter consolidation.

Jane: They show this method improves preference accuracy on preference data and reduces reward hacking during PPO training compared to standard dense RMs.

Conclusion: Tom: So wrapping up this discussion on "UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hacking." The authors are Lingling Fu and Yongfu Xue from Guangxi University.

Jane: The implication is that this approach gives developers a more reliable way to build reward models without having to jump into complex MoE inference or managing large ensembles.

Lu: It’s about creating a practical system where you get the nuanced preference capture from experts but you don't pay the price in runtime complexity for every single interaction.

Meng: For someone building an AI application, this means they can rely on this reward model to guide their policy optimization with greater confidence that it’s actually moving toward human-preferred outputs.

Tom: It suggests that using a structured approach to build reward models, like upcycle and merge, is a better way than just relying on the simpler dense models or even the SFT checkpoint alone.

Jane: It moves the conversation toward making reward systems more stable and less susceptible to those kinds of adversarial training issues we discussed earlier.

More episodes

← Home