2608.09324-CoRE: Consensus Rewards via Equilibrium for Test-Time Reinforcement Learning

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "CoRE: Consensus Rewards via Equilibrium for Test-Time Reinforcement Learning".

Jane: The paper was written by Ambuj Mehrish and Sebastiano Vascon from Ca' Foscari University of Venice.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: So we have a paper from Ca' Foscari University of Venice, by Ambuj Mehrish and Sebastiano Vascon. It's about test-time reinforcement learning, which is what happens when you let a language model keep training on a test set without any ground-truth labels. The standard trick is to sample many answers, take a majority vote, and reward the model for matching that vote. The authors argue that the vote is a brittle way to extract supervision, and they replace it with a method they call CoRE.

Jane: And it reads the same rollouts differently, rather than sampling more or adding an auxiliary model?

Tom: Exactly. The rollouts become a graph, where edges connect answers that agree, and the edge weights also look at how similar the reasoning is and how confident the model was. Then a game-theory procedure called replicator dynamics finds the dominant set of mutually supporting trajectories. From that single equilibrium you get a refined pseudo-label, a graded reward for each rollout, and a gate that decides whether a question is even worth training on.

Lu: The numbers back it up. Across seven backbones and five benchmarks — 42 model-benchmark cells with three seeds each — CoRE improves the untrained base by 21 point 7 points on average, compared with 20 point 4 for majority-vote TTRL. It beats the vote by up to 7 point 5 points when answers are contested, and it wins wherever agreement is contestable while staying inside seed noise where the majority is reliable.

Meng: What I like is that it costs nothing extra. You generate the same 64 rollouts you already needed and read them more carefully. No added model, no labels, and no extra sampling — the graph and the equilibrium are just a smarter use of what you already have.

Lalam: And here's the bigger picture. The whole training loop hangs on that consensus function, because it's the only supervision the model gets. If the vote picks a wrong label, you are actively rewarding wrong answers. Showing that the consensus operator is itself a real design choice, with theory behind it, changes how you would build these adaptation loops.

Jane: It also reaches the same final accuracy as voting in 54 to 70 percent fewer training steps. That matters if you're paying for compute.

Meng: And everything stays self-supervised — ground truth only shows up in the final evaluation, never in the training loop.

Tom: Right, and the analysis shows majority voting comes back as a special case, so switching costs very little. But the paper opens with a genuine puzzle about voting that's worth sitting with for a minute.

Page 1 of the paper: Tom: Where we left off, the whole idea is that a consensus function decides what test-time RL can learn. Page one shows why the default, majority voting, is a weak choice. There's a paradox first, and it's genuinely fun: a model trained on majority votes can end up more accurate than the vote itself.

Jane: How does that not collapse everything? If the supervision is wrong more often than the model, you'd expect the model to get worse.

Tom: The answer is that model errors tend to scatter. When a question is hard, the model fails in many different ways, so the wrong rollouts disagree with each other. Even when the majority label is wrong, most incorrect rollouts vote against it and get penalized. That partial negative signal preserves something worth learning from.

Lu: But the paper identifies the failure mode where that breaks. They call it concentrated error. A systematic mistake becomes the plurality while the correct derivation stays in the minority. Then the vote rewards the wrong trajectories and penalizes the right ones, and updating on that amplifies the error across steps.

Meng: There are two more problems with a vote. A binary reward gives the same score to a well-supported derivation and a lucky guess. And every question contributes equally, whether its rollouts form a tight consensus or a fragmented set of incompatible solutions.

Jane: Because a vote compresses each trajectory down to its final answer and each answer class down to a count. You lose the model's confidence during generation, and you lose the relational structure among reasoning paths entirely.

Lalam: So voting can't even see the difference between a cluster of solutions that agree on substance and a bunch of random guesses that happen to land on one number. The paper runs the math to show you can't fix this by counting more carefully — graph structure alone rarely beats the vote, and confidence weighting alone is also limited.

Tom: Exactly, and the two signals turn out to be complementary. That complementarity becomes the entire design of the method. It also has a fairly deep ancestry in graph clustering and game theory, which is what the next page covers.

Page 2 of the paper: Tom: Page three is where the paper lays out the intellectual toolkit. The contributions are threefold: a method, a theory for when it works, and evidence across those 42 settings. But the lineage is the interesting part — there's a whole body of work on aggregating multiple samples at inference time.

Jane: Self-consistency, from the chain-of-thought literature. You sample many reasoning paths and take the majority. The paper cites a confidence-weighted variant, CISC, which cuts the number of paths needed by more than 40 percent.

Lu: But that line operates at inference only. You pick the best answer and move on. A parallel line uses learned verifiers to score or rerank samples. CoRE's shift is that it turns the aggregation itself into a training signal for reinforcement learning.

Meng: And the clustering machinery comes from an older place. A theorem by Motzkin and Straus relates maximum cliques to maximizing a quadratic function over a simplex. Then Pavan and Pelillo generalized that idea to weighted graphs, under the name dominant sets — basically maximal cliques where the weights are allowed to vary.

Lalam: The algorithm that finds them is replicator dynamics, which borrows from evolutionary game theory. You have a population of strategies, and the ones with higher payoff grow over time. A result called the Baum-Eagon inequality guarantees this iteration keeps improving at every single step.

Tom: The paper's claim is that CoRE is the first method to use dominant-set extraction for reinforcement learning rewards. That's a genuinely new borrowing, and it fits — because a dominant set is exactly a group of trajectories that mutually support each other rather than just agreeing on a label.

Jane: And there's a subtle design choice on the RL side. GRPO normalizes rewards within a group, so any per-question scaling of rewards gets cancelled out. That's why the cohesiveness gate gets applied to the loss instead of the reward.

Lu: It also means that when the gate is always one, you recover the standard objective. So someone who wants to bolt this onto an existing pipeline doesn't have to change anything else.

Meng: So the ingredients are clear. The next page shows how you actually turn 64 raw rollouts into a graph, weight the edges, and run the dynamics.

Page 3 of the paper: Tom: We know the ingredients now; the construction is on this page. Each question gives you 64 rollouts, and those become the nodes of a graph. An edge exists between two rollouts only if they give the same final answer — symbolic equivalence, not string matching. That's the hard gate that structures everything.

Jane: But the edge weight also depends on how similar the reasoning is. They use TF-IDF vectors over character n-grams and take the cosine similarity between the reasoning texts.

Tom: Right, and that similarity only applies within the same answer class. Different answers share no edge, no matter how similar the prose looks. The formula combines the answer match with a weighted reasoning term, and a floor parameter at 0 point 1 guarantees agreeing rollouts keep a minimum connection even when their derivations are lexically very different.

Lu: Then confidence enters. Each rollout gets its mean token log-probability as a confidence score, those get converted into relative weights anchored at the most confident rollout, and the temperature is set to 0 point 25. Each edge is scaled by the geometric mean of its two endpoints' weights.

Meng: Which preserves the block structure and makes sure an edge only survives when both endpoints are confident. There's also a diagonal penalty so that a single rollout can't be chosen as the consensus on its own — a singleton has a negative internal score.

Jane: The dynamics then start from the center of the simplex, giving every node an equal chance, and iterate the replicator equations until convergence. Three lemmas guarantee this is sound: the constant shift doesn't change the maximizers, the objective rises monotonically, and no singleton can win when a genuine clique exists.

Lalam: The readouts are what feed the training. The answer class holding the most equilibrium mass becomes the pseudo-label. Each rollout gets a graded reward proportional to its affinity to that consensus, so a well-supported derivation scores higher than a marginal one. And the gate measures the overall coherence of the consensus — how much the group really agrees.

Tom: One subtlety: the rewards use the uncalibrated affinity, not the confidence-weighted one, because confidence has already shaped the equilibrium. Using it again would double-count.

Lu: And the whole thing is a strict generalization of the vote. Set kappa to one and make confidence uniform, and the extracted class is exactly the plurality.

Meng: So the recipe is complete. The question is whether it actually helps, and the first big answer comes in the experimental table for the math-specialized models.

Page 4 of the paper: Tom: We have the construction and the theory, so this page asks the empirical question. The math-specialized table covers Qwen2 point 5-Math at 1 point 5 and 7 billion parameters, plus DeepSeek-Math at 7 billion, each on six benchmarks. The family average improvement over the no-RL base is 21 point 6 points for CoRE, against 19 point 9 for majority voting and 19 point 8 for the graph-only variant.

Jane: So the gap is real but not huge — which fits the theory that CoRE wins where consensus is contestable, not everywhere. And notice the graph-only arm is no better than the vote. That matches the prediction that structure alone doesn't rescue a minority.

Lu: The individual cells tell the story. On Qwen2 point 5-Math-1 point 5B, CoRE beats voting by 5 point 3 points on MATH-500 and 5 point 0 on MATH Level 4. On the 7B model the biggest margin is 7 point 5 points on GPQA. DeepSeek-Math gains 3 point 8 points on AMC.

Meng: But look at where it doesn't win. On MATH Level 5 for the 1 point 5B model, and on MATH-500 for both 7B models, CoRE trails by at most 0 point 3 points. That's inside the seed noise, and those are settings where majority voting already sits above 85 percent or around 50, with little recoverable headroom.

Jane: There's also the out-of-domain GPQA column. For the 1 point 5B model, GPQA is both a different subject and a different format — multiple choice with four options — and the model hovers near the 25 percent random-guess floor. The paper is honest about this: the benchmark is a probe of the operating envelope, rather than an in-domain evaluation.

Lalam: The consistent pattern is what matters. The gains concentrate where a coherent, confident correct minority exists to be recovered. They vanish where the vote is already right, or where nobody in the sample actually knows the answer.

Tom: And the paper ties this back to the theory rather than leaving it as anecdote. The measured wins and losses line up with the recovery threshold from the earlier analysis. That's why the next table, with vanilla and instruct models, is a real test rather than a formality.

Page 5 of the paper: Tom: The math-specialized results looked strong, and now the instruct models put that to a harder test. On LLaMA-3 point 1-8B, CoRE beats majority voting by 4 point 2 points on MATH Level 4 and 5 point 1 points on GPQA. On Mistral-Nemo-Instruct, the margins are 2 point 2 points on MATH Level 5 and 3 point 8 on GPQA, with a family average of 19 point 1 points over the base.

Jane: What strikes me is that these are the noisier backbones. Majority voting on Mistral actually hurts accuracy on several benchmarks compared with doing nothing. That's the known failure mode of self-training — a weak model locks onto systematic errors. CoRE still manages to extract signal, because it can find a coherent correct cluster even when the plurality is wrong.

Lu: The vanilla models in the same family line up too. Qwen2 point 5-7B and Qwen3-8B in non-thinking mode show a mean improvement of 24 point 5 points, with CoRE beating the vote by 6 point 2 points on MATH Level 5

Page 6 of the paper: Tom: We've been exploring how CoRE replaces the majority ballot with a graph-based consensus, and page eleven steps back to show where that machinery actually earns its keep.

Jane: It lays out three operating regimes, and the first one is blunt: at the competence floor, there's simply nothing to recover. Weak models on out-of-domain or competition-level problems produce roll-outs with no coherent structure at all, so CoRE and majority voting both just sit at the baseline.

Tom: Right, that's the 1 point 5B math model on GPQA, or Mistral-Nemo on eyeME. When nobody in the sample can solve the question, no consensus rule can manufacture signal from noise.

Jane: At the opposite extreme, you get saturation. A strong model produces nearly unanimous roll-outs, the majority ratio approaches one, and CoRE's pseudo-label ends up matching the vote anyway.

Tom: So it gracefully degrades to majority voting exactly when majority voting is already fine. The interesting territory is the middle, where a coherent, confident correct minority exists but gets outvoted.

Jane: And that's where the two signals split the workload. Confidence weighting rescues cases where the wrong plurality is a degenerate cluster, like repeated empty boxed tokens, while the graph structure handles the harder cases where the wrong answer is fluent, popular, and confidently stated.

Tom: Page eleven also flags a genuine limitation. Confidence only helps if correct clusters are, on average, more confident than incorrect ones. If that ordering is weak or reversed, confidence adds nothing.

Jane: Which is why the method combines both signals instead of betting on one. Each one becomes informative in a different part of the envelope, and when no recoverable minority exists, CoRE just behaves like the old ballot.

Tom: So the consensus operator adapts to the situation rather than forcing one rule everywhere. That naturally raises the question of what happens when both signals fail together.

Jane: And that's exactly what the qualitative examples in the appendix dig into, with a concrete geometry question where the graph decisively beats all three baselines.

Conclusion: Jane: We set out to question the majority vote as the default consensus rule for test-time RL, and CoRE answers that by treating roll-outs as a graph and extracting a dominant set through replicator dynamics.

Tom: That single design choice gives you three upgrades over the ballot box: a pseudo-label that can overturn an incorrect plurality, a graded reward instead of a binary one, and a cohesiveness gate that down-weights fragmented questions.

Jane: The theory makes it falsifiable. Majority voting is a special case, the extraction threshold has a sharp formula, and confidence lowers that threshold multiplicatively in nats.

Tom: And the experiments match the predictions, not just the averages. The gains land on contested high-disagreement benchmarks, vanish at the competence floor, and dissolve into seed noise under saturation.

Jane: The practical angle is what wins me over. Same 64 roll-outs, no auxiliary models, no extra sampling, just a smarter read of what you already generated. Plus the sample efficiency gain, reaching the vote's plateau in about half the steps.

Tom: The deeper implication is that the consensus operator is a real design choice, not a plumbing detail. If the only supervision in adaptation comes from self-agreement, then how you define that agreement sets the ceiling on what the model can learn.

Jane: It also opens a path for self-training beyond math. The authors mention hidden-state representations instead of lexical features, and long-context reasoning models as natural extensions.

Tom: And honestly, the fact that the recovery examples include a geometry question where only the graph structure finds the correct minority, while confidence alone fails, shows the two signals are genuinely complementary rather than redundant.

Jane: We should also note the transparency: the codebase and fixed hyperparameters are released, so the whole thing is reproducible from the repository.

Tom: That's a good place to leave CoRE. We've seen how equilibrium-based consensus beats the ballot, and the next paper on our list tackles a different angle on reward design for reasoning models.

Jane: Let's turn the page and see what's coming up.