2608.07180-Momba: Network Modernization Improves Multi-Objective Reinforcement Learning

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Momba: Network Modernization Improves Multi-Objective Reinforcement Learning".

Jane: The paper was written by Adam Štafa, Santeri Heiskanen, Petr Novotný and Joni Pajarinen from .

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: So we're looking at a paper from Masaryk University in Brno and Aalto University in Finland, and the title makes a bold claim — modernizing the network, not the algorithm, is what improves multi-objective reinforcement learning. The name Momba is playful, but the argument behind it is very pointed. I think the title is the whole thesis in seven words.

Jane: It really is. In single-objective RL, there's been a wave of work showing you can get enormous gains just by changing the neural network architecture — normalization, better critics, residual connections — while keeping the learning rule untouched. And this group looks at the multi-objective side and says, why is that wave missing here? They treat the field's favorite algorithms as fixed and ask what the network is costing them.

Lu: The author mix tells part of the story. Adam Štafa and Petr Novotný work on the formal and theoretical side at Masaryk, while Santeri Heiskanen and Joni Pajarinen bring the Aalto robotics and machine learning perspective. You can see both flavors — rigorous benchmarking and careful ablation — all through the paper. That combination matters because the claim is empirical, so the experimental discipline has to be solid.

Meng: And that discipline is exactly what the title demands. They're not proposing a new way to handle multiple objectives. They take an existing algorithm called CAPQL and give its function approximators a serious upgrade, and the performance jumps. So "modernization" is doing a lot of work in that title.

Tom: Right, and it's a provocation to a field that has spent years on clever preference sampling and specialized update rules. This paper says the bottleneck might be much more mundane — simple feedforward networks that aren't expressive enough to represent value functions across different trade-offs. If they're right, a lot of recent MORL machinery has been polishing the wrong part of the pipeline.

Lalam: What I find exciting is the wider implication. If architecture modernization transfers from single-objective to multi-objective RL this cleanly, then a whole family of problems — robot control, treatment planning — could get better without any new theory, just by borrowing what deep RL already learned about building networks. That's a cheap win for a lot of application areas.

Jane: The title "modernization" is exactly the right word, then. It's like upgrading the engine instead of changing the route you drive. And since the authors are building directly on SimbaV2, a recent single-objective architecture, the next question is what exactly they borrowed and what they had to adapt.

Tom: That's our next stop — the paper's summary, and the core question of why multi-objective RL has been leaving performance on the table.

Summary: Tom: We've established the provocation — architecture over algorithm. Now the paper's summary lays out the problem clearly: multi-objective RL wants a set of policies that balance conflicting objectives, and most methods handle that by conditioning a single policy on a preference vector, which is the trade-off knob. That framing immediately tells you why representation matters.

Jane: And the key observation is that even though the optimal policy can look very different depending on the trade-off, the standard choice is still a plain feedforward network conditioned on that preference. You're asking one network to cover the entire Pareto front, and then you give it a fairly weak function approximator to do that job. That mismatch is where the paper starts.

Lu: Right, and they connect this to a real gap in the literature. Single-objective RL has a string of recent papers showing that normalization and distributional critics improve sample efficiency and final performance, with analyses of why they help — better conditioning of the optimization, less overfitting to early data. The multi-objective side kept its simple feedforward networks, and this paper is essentially importing that whole toolbox into a field that hasn't touched it.

Meng: So what they do is take an entropy-regularized MORL algorithm called CAPQL, which is essentially the multi-objective cousin of Soft Actor-Critic, and they bolt on three things from the recent single-objective toolbox: observation and feature normalization, weight normalization, and a distributional critic that models the distribution of returns instead of just their expected value. That last part turns out to be the most interesting.

Tom: And the punchline of the summary is that these changes substantially improve the quality of the solution sets without requiring major changes to the underlying algorithm. That's a strong statement. It means the gains come from representation, not from new learning rules.

Jane: I also like that they framed it as an open question first — could MORL benefit from these advances? — and then actually tested it. A lot of papers assume the answer before running the experiments. The summary is refreshingly honest about what they did and what they found.

Lalam: There's a bigger pattern here. In deep RL, we keep rediscovering that how you parameterize the problem matters as much as the objective you optimize. This paper extends that lesson to a field that was overdue for it, and if the gains hold across tasks, it changes where researchers should spend their effort.

Tom: And that leads us to the actual improvements — the three components, and especially the distributional critic, which needed some clever adaptation to work in the multi-objective setting.

Improvements: Tom: We've seen why architecture was the missing piece — now let's get concrete about what the modernization actually contains. The first two pieces are normalizations: they normalize observations with running statistics, they normalize hidden features onto a hypersphere, and they normalize the weights after every gradient update. All of that keeps training stable and prevents the network from overfitting to early experiences.

Jane: And the third piece is the distributional critic. Instead of predicting the expected return, the critic predicts a full distribution over returns, which is known to stabilize learning and improve the conditioning of the optimization. But there's a catch: in multi-objective RL, returns are vectors, and categorical distributional RL was designed for scalar returns.

Lu: That's where the paper's most interesting idea comes in. They notice that the policy only ever sees the critic through a scalarized value — the preference vector dotted with the Q-values. So instead of modeling the multivariate joint distribution of returns, like some earlier work attempted, they model the distribution of the scalarized return directly. One conditioned univariate distribution per preference.

Meng: It's a simpler adaptation, and honestly a pragmatic one. Previous attempts at multivariate distributional RL required kernel functions to measure distance between distributions, or were limited to tabular settings. By targeting linear scalarization — which is already the common assumption in MORL — they sidestep all of that complexity. And in the appendix they show that if you need the non-scalarized vector returns, you can learn the marginal distribution of each component separately, and it performs just as well.

Tom: There's also a practical detail: they normalize each reward component by the maximum return seen so far during training. That matters because different preferences can produce very different return scales, and the categorical critic needs a fixed support interval to work well.

Jane: They also changed how preferences are sampled — once per episode from a uniform distribution, instead of at every timestep like the original CAPQL. That puts them in line with most other MORL work, which is smart because it means the gains can't be credited to a fancy preference selection trick.

Lalam: All together, that's a real philosophy difference. The field has been trying to solve multi-objective RL with better search over preferences. This paper says, give the network a better internal representation and the search problem gets easier on its own.

Tom: And the obvious question now is whether it actually worked. So in the next segment, we look at the first page and the empirical evidence — and the numbers are pretty dramatic.

First page: Tom: We've walked through the three components and the clever scalarized critic. Now the first page of the paper formalizes all of that into two contributions: first, expressive architectures with a distributional critic substantially improve MORL performance without complex preference selection or MORL-specific update rules, and second, a simple adaptation of the categorical critic to learn scalarized return distributions. Both claims are testable, and they test them thoroughly.

Jane: And those two contributions aren't just claims — the results section backs them up. Aggregated over seven continuous control tasks, Momba gets roughly 35 percent higher hypervolume and 16 percent higher expected utility than PGMORL, the runner-up. Against CAPQL, the algorithm it's built on, that's about 132 percent and 35 percent improvement respectively.

Lu: Those are large gaps, especially the comparison to its own backbone. And I appreciate that they report both metrics because hypervolume captures convergence and coverage while EUM captures actual user utility. The two metrics agreeing makes the result more convincing.

Meng: The sample efficiency story is just as strong. In Ant and Humanoid, Momba reaches PGMORL's final performance after only 100,000 and 200,000 steps, while PGMORL was trained for tens of millions of steps. And when they match update-to-data ratios against GPI-LS, the method famous for sample efficiency, Momba beats it using plain uniform preference sampling.

Tom: Then the ablations show where the gains come from. The distributional critic is the most important component — even a plain MLP with the categorical loss matches or beats the fancy Simba architecture with a standard MSE critic. And among the normalizations, observation normalization contributes the most, around 55 percent according to their Shapley value analysis.

Jane: That ablation is the most convincing part of the paper for me. They systematically varied every component, and they showed that parameter count alone doesn't explain the gains — bigger MLPs with the old loss stay bad. It's the combination of representation and loss that unlocks the performance.

Lalam: The broader implication is that a lot of the sophisticated machinery in recent MORL papers — the preference prioritization, the self-consistency losses — may be solving a symptom rather than the cause. If the function approximator is the limiting factor, architecture research deserves a much bigger share of attention in this field.

Tom: And that sets up our final segment — what this means going forward, where the approach falls short, and what we should watch for next.

Conclusion: Tom: So let's wrap this up. The paper took a standard multi-objective algorithm, gave it a modern network with normalization and a distributional critic, and got dramatic gains in both final performance and sample efficiency across continuous control benchmarks. If you work in MORL, you don't need to adopt a new learning rule here — you need a better network under the hood. That message is simple and it travels well.

Jane: And the analysis holds up. The distributional critic carries most of the weight, observation normalization comes second, and the whole thing works without fancy preference selection. That's a clean result, and one that practitioners can pick up almost immediately. I think this will show up in a lot of future MORL codebases.

Lu: But the authors are honest about the limits. They only consider linear scalarization, which is the common assumption in MORL but doesn't cover every setting, like nonlinear or learned scalarization. And the benchmarks are continuous control tasks, mostly with two objectives — only one environment has three. So we don't know yet how this scales to many objectives or to discrete state and action spaces. That's a real open question, and I'd love to see someone run those experiments.

Meng: Still, the practical path is wide open. Anyone using SAC-style methods can take these changes almost off the shelf, and the vector version of the distributional critic widens the reach even further. This is the kind of paper where you read the appendix and immediately start thinking about which of your own problems could benefit from the same treatment.

Lalam: For me, the lasting contribution is that the field should stop treating architectures as an afterthought. This paper doesn't just improve one algorithm — it opens a research direction of modernizing all the existing MORL methods that were built on older network designs. That could shift where a lot of research effort goes over the next few years.

Tom: And that's a great note to end on. The rigorous ablations, the honest limitations, the genuinely practical payoff — this made for a really satisfying discussion. Thanks everyone for the conversation, and we're ready to move on to the next paper.

Jane: Goodbye from all of us, and see you on the next episode.