2608.07424-CoBa: Cost-Effective Test-Time Scaling via Compute-Balanced Routing

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "CoBa: Cost-Effective Test-Time Scaling via Compute-Balanced Routing".

Jane: The paper was written by Yan Zhou, Yue Ouyang, Kaiyang Zheng and Suncheng Xiang from Changsha University of Science and Technology and Shanghai Jiao Tong University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: We finally get to sit down with this one, and honestly the framing grabbed me on the first read. Most test-time scaling work treats sampling more, thinking longer, and evaluating harder as separate knobs, and this paper says they all draw from the same fixed inference budget, so they compete.

Jane: Then the move is to treat that competition as an allocation problem. The system has to decide whether the next unit of compute goes to generating another candidate, checking a candidate with a cheap judge, running a strong verifier, or stopping altogether. That last action — stopping — is the one most test-time scaling papers forget to include.

Lu: They formalize it nicely, too. There's a state made of the candidates collected, the verifier scores, the remaining budget, and the history, and an action space spanning sampling, verification at different strengths, and stopping. Everything the policy knows is right there in the state vector.

Meng: And the cost metric they lean on is parameter-weighted tokens — each model's token count multiplied by its size in billions. A token from a 14B model is simply more expensive than one from an 8B model, and accuracy-only tables never show that.

Lalam: The headline result follows from that. Their strongest variant hits 85 point 13 percent macro accuracy, statistically matching a self-evaluation weighted-voting proxy at 85 point 20 percent, but using 49 point 1 percent fewer parameter-weighted tokens. Against best-of-16 majority voting it lands within 0 point 01 accuracy points while using 58 point 9 percent fewer.

Tom: Those numbers come from 3,129 example-generator evaluations across MATH-500, eyeME 2024 and 2025, AMC 2023, and the hard Reasoning Gym subset. Three local generators — Qwen3-14B, Phi-4-reasoning, and Qwen3-8B — all replayed over shared candidate pools.

Jane: So the claim is that you can reach the accuracy of heavy sampling or broad self-evaluation just by being selective about which candidates deserve the expensive verification. Cheap evidence first, strong verification only where it can change the decision. That's the whole method in one sentence.

Meng: But the paper doesn't oversell, and I respect that. Best-of-16 keeps a small paired edge of about 0 point 70 points, and it pays for that edge with 2 point 43 times the parameter-weighted cost. So the result is a Pareto improvement over moderate baselines and a cost cut at the high-accuracy end.

Lu: Then there's the oracle gap to keep in mind. The pool oracle sits at 91 point 36 percent, so plenty of headroom remains. Sometimes the correct answer sits in the pool and routing misses it; sometimes the pool simply contains no right answer.

Lalam: And that distinction ends up being the most productive part of the paper. If the answer exists but goes unselected, the next unit of compute should sharpen verification. If it doesn't exist, the next unit should expand generation. The oracle gap becomes a design signal instead of just a score.

Tom: Good place to start digging in, then. The first page sets up the central hypothesis that accuracy gains and cost savings have to be evaluated together, not as separate virtues.

Page 1 of the paper: Jane: Page one opens with exactly that hypothesis, and the supporting argument is simple. A system that always samples sixteen candidates or always runs a strong evaluator can be accurate, but it spends the same expensive actions on problems that were already settled.

Tom: "Settled and ambiguous examples" — that phrase carries the paper. Once sampled answers agree and the cheap judge is confident, extra compute is just going through the motions. Routing exists to eliminate that waste.

Lu: Page one also establishes the control-loop picture. The figure shows a state vector, a budget tracker, cheap evidence signals, and an action router all connected. It's a controller wrapped around the language model, deciding what happens next.

Meng: And the contributions listed there — formalizing test-time reasoning as allocation, introducing a reproducible routing policy with light and strong verification tiers, and running controlled replay experiments — the replay piece is the one that earns trust.

Lalam: The replay protocol means every method sees the same sixteen generated candidates per problem. Routing can change which candidates get inspected and selected, but it can't invent new generations. That keeps the comparison honest.

Tom: The abstract also gives the clearest statement of the method anywhere in the paper. Get a small candidate set, apply cheap verification broadly, then route uncertain or high-value candidates to stronger verification. Cheap broadly, strong narrowly.

Jane: And they're explicit that they are not proposing a new generator or a new verifier. The models are the same ones the baselines use. CoBa only changes when those models get called — that's the entire intervention.

Lu: That's the cleanest experimental design you could ask for. If accuracy holds while cost drops, the gain has to come from the allocation policy, not from a cleverer model. The ablation structure on later pages confirms that.

Meng: The evaluation scope is visible on this page as well — three locally served generators on competition math and procedural symbolic reasoning. No closed APIs, no unavailable checkpoints anywhere in the stack.

Lalam: So the whole paper runs on hardware a decent local setup can reproduce. That's increasingly rare in this corner of the literature, and it makes the numbers worth taking seriously.

Tom: Then page two positions the work against the field — the sampling line, the verification line, the adaptive compute line — and shows what's missing from each of them.

Page 2 of the paper: Jane: Page two runs through that landscape with a clear pattern in mind. Self-consistency buys diversity, chain-of-thought buys reasoning depth, verifiers buy selection — each family treated as its own recipe, its own scaling knob.

Tom: The gap the paper identifies is that nobody treats those as competing draws on one budget. Under a fixed inference budget, spending the next token on sampling means not spending it on verification. That's the hole the routing framework fills.

Lu: There's also a current and practical discussion of verifier cost becoming a first-order variable. With generative process reward models and evaluation-time scaling using reasoning models as judges, strong verification is no longer cheap. Deciding who gets verified is a real economic decision.

Meng: The adaptive allocation line — FrugalGPT, RouteLLM, early-stopping monitors like interwhen — shares the intuition that inputs deserve different amounts of compute. CoBa applies that intuition inside a reasoning workflow, routing among sampling, verification, and stopping rather than between whole models.

Lalam: The positioning is pointed without being dismissive. The section concedes that each method family does something real — diversity, selection, waste avoidance — and then says the contribution is treating those as complementary actions under one controller.

Tom: The surveys of test-time scaling get cited here too, organizing the field by what, how, and where compute gets scaled. CoBa's answer is essentially "at the action level, guided by cheap evidence."

Jane: The end of page two starts the formal setup, and that's where the definitions land. State, action space, and the two cost metrics — total tokens and parameter-weighted tokens — all appear there.

Meng: Right, and that parameter-weighted metric is the one most papers ignore. A 14B token is not the same cost as an 8B token, and any allocation policy that treats them identically is blind to the thing it's supposed to optimize.

Lu: The objective function on that page has a sensitivity parameter lambda that trades accuracy against budget. The experiments use fixed replay policies, but the framework itself has a tuning dial.

Lalam: So by the end of page two, the machinery is in place. Page three then makes everything concrete — the routing algorithm, the stop criterion, the scoring rule, and the three variants that trace the cost-accuracy curve.

Page 3 of the paper: Tom: Page three delivers Algorithm 1, and the structure is almost elegant in its simplicity. Warm up with two candidates, score all of them with the cheap frequency check and the lightweight judge, then enter the decision loop.

Jane: The loop is where the adaptivity lives. The policy computes answer agreement, score gap, and uncertainty. If the top answer holds at least 60 percent agreement and the lightweight judge scores it at least 0 point 7, the system stops; otherwise it generates one more candidate and scores it cheaply.

Lu: That stop criterion is fully transparent, and that matters. It's not a learned function hidden inside a network; it's literally "the answers agree and the cheap judge is confident, so we're done." Anyone can audit it against the logs.

Meng: The final ranking score is explicit as well. Answer frequency carries weight 0 point 20, the 8B judge score carries 0 point 30, the optional process-verifier score carries 0 point 15 when it parses, and the strong 14B deep verifier carries 0 point 45.

Lalam: The weight distribution tells the story. The strongest verifier has the biggest weight, but it only touches the top-K routed candidates. Cheap evidence decides who gets routed; strong evidence decides among the routed few.

Jane: One detail that matters here is the renormalization when process-verifier scores are missing. Unrouted candidates don't get an artificial penalty just because they skipped the strong verifier. The weights renormalize, so absent evidence is never misread as a negative signal.

Tom: The routing variants trace a clean curve from that base. Light runs just the two-candidate warm-up with no extra sampling. Balanced expands to four candidates and routes two to the strong verifier. Strong expands to eight and routes four.

Lu: And the table on that page locks those values in before final aggregation, which is a real commitment. Test labels stay reserved for evaluation, so the thresholds can't be accused of being cherry-picked.

Meng: Then the offline replay protocol makes every method comparable. Sixteen candidates per example, same pools for everyone. CoBa chooses prefixes, subsets, and routed verification calls, but nothing beyond what the baselines could also see.

Lalam: That's what makes the later cost comparisons meaningful. When the paper reports a 49 percent token reduction at matching accuracy, both methods drew from the same generated evidence, so the difference is allocation.

Tom: With that foundation set, page four fills in the experimental machinery — datasets, models, baseline taxonomy, metrics — all the choices that give those numbers their credibility.

Page 4 of the paper: Jane: Page four introduces the benchmark suite in full. Five test sets — MATH-500, eyeME 2024, eyeME 2025, AMC 2023, and the hard Reasoning Gym subset — giving 1,043 unique examples and 3,129 example-generator evaluations across fifteen pairs of datasets and generators.

Tom: The hardware story is relatable, too. Local RTX 3090 GPUs, FP16 or quantized serving, vLLM for OpenAI-compatible serving. That's a modest setup, and it means the results reflect what many practitioners actually run.

Lu: The model lineup is fixed across the whole study. Qwen3-14B and Qwen3-8B generate, Phi-4-reasoning generates as well, the 8B model acts as the lightweight judge, and the 14B model acts as the strong outcome verifier. Phi-4-reasoning also supplies auxiliary process-verifier scores.

Meng: And the paper is upfront that those process-verifier scores were sparse in the final run, so the main comparison centers on outcome verification. It's an honest note — they don't lean on a score stream that never fully materialized.

Jane: The baseline taxonomy on this page is one of the most valuable parts. Direct baselines like best-of-N and self-consistency run literally over the shared pool. Local proxies preserve the inference pattern of recent methods but get labeled as proxies. The pool oracle stands apart as an upper bound that can see correctness.

Tom: That taxonomy matters because a lot of recent test-time scaling results depend on private checkpoints or closed evaluator APIs. Anchoring the comparison to locally reproducible methods is what makes the frontier movement credible.

Lu: The metrics list includes measured latency in seconds alongside accuracy, total tokens, model calls, and parameter-weighted tokens. The cost picture has several independent dimensions rather than one convenient proxy.

Meng: Answer extraction also gets a rigorous treatment — boxed answers, final-answer statements, normalization of fractions, tuples, and π. Because routing changes which candidates survive to the end, extraction has to be identical across methods or the accuracy comparison breaks.

Lalam: By the end of page four, every methodological choice is on the table. Page five then delivers the results, and the headline numbers genuinely deliver on the abstract's promises.

Page 5 of the paper: Tom: Page five carries the main results table, and the numbers land exactly where the abstract said they would. CoBa-Routed-Strong reaches 85 point 13 percent macro accuracy at roughly 58,000 total tokens and 630,000 parameter-weighted tokens per item.

Jane: The comparison pairs are the ones to focus on. The routed system is statistically indistinguishable from the self-evaluation weighted-voting proxy at 85 point 20 percent, with 49 point 1 percent fewer parameter-weighted tokens. It also lands within 0 point 01 accuracy points of best-of-16 majority voting, with 58 point 9 percent fewer.

Lu: Figure two shows the same result as a frontier. The routed variants occupy the upper-middle region — light is inexpensive, balanced beats best-of-4 at similar cost, and strong reaches the accuracy zone of best-of-16 and self-evaluation while spending far less.

Meng: But the per-dataset table is where the nuance jumps out. eyeME 2024 goes from 65 point 6 percent greedy to 82 point 2 percent routed — a massive allocation gain. eyeME 2025 only reaches 71 point 1 percent against an oracle at 83 point 3 percent.

Jane: That contrast between the two eyeME years is the most instructive thing in the results. On eyeME 2024, extra candidates and routed verification often recover from a weak first sample. On eyeME 2025, the correct answer often isn't in the pool at all, or the verification signals can't surface it.

Tom: The paper splits those into two failure modes — allocation errors, where better routing could pick an existing correct answer, and generation errors, where the next useful unit of compute should create better candidates before scoring deeper. Deeper evaluation on a wrong pool mostly audits bad candidates more carefully.

Lu: Reasoning Gym tells a similar story from the other direction. CoBa reaches 92 point 3 percent against an oracle of 99 point 8 percent. The pool is rich with correct answers, and the policy still can't always identify the right one.

Meng: The deployment read is that the savings spread across the benchmark. Easy examples stop after cheap agreement; hard contest problems consume extra samples and strong-verifier calls. Uniform best-of-N pays the hard-example budget on every single problem.

Lalam: That's the whole routing thesis in one sentence — concentrate the budget where the decision can still change, rather than spending everywhere identically. Page five backs that thesis with hard numbers.

Tom: Then page six goes one layer deeper, showing the actual action mix across datasets and running the significance tests that separate real gains from noise.

Page 6 of the paper: Jane: Page six opens with the behavioral evidence for adaptive allocation. The action mix per dataset is genuinely different — MATH-500 stops early after lightweight judging, while the contest sets trigger more sampling and more strong-verifier calls.

Tom: That's the policy's fingerprint. The system isn't applying a uniform compute cap; it's shifting the composition of compute based on what the cheap evidence says about each problem.

Lu: The routing-strength ablation shows the expected progression. Light gets 78 point 88 percent, balanced gets 82 point 92 percent, strong gets 85 point 13 percent. And the jump from light to balanced is the biggest, which tells you the first routed candidates carry the most value.

Meng: The paired bootstrap tests then do the statistical work. CoBa-Routed-Strong beats greedy by 3 point 74 points with a 95 percent confidence interval from about 2 point 97 to 4 point 54. Against the self-evaluation proxy, the difference is minus 0 point 16 points and the interval includes zero — statistically indistinguishable.

Jane: And the best-of-16 comparison stays honest. Best-of-16 holds a 0 point 70 point edge, with a confidence interval from minus 1 point 25 to minus 0 point 16, but it pays 2 point 43 times the parameter-weighted cost. That's a real advantage, and a very expensive one.

Tom: There's also a result most papers would quietly drop — the learned MLP controller degenerated to a near-greedy policy in the leave-one-dataset-out setting. They present it as a negative result and say learning a robust controller from offline trajectories remains open.

Lalam: Publishing that failure is a genuine scientific strength. It keeps the claims honest: the routing gains come from the transparent hand-designed policy, not from a trained black box that might be overfitting the benchmark.

Lu: The cost table at the end of the page frames the high-accuracy regime cleanly. The three high-accuracy methods all sit around 85 percent, but their relative parameter-weighted costs are 1 point 0, 1 point 96, and 2 point 43 times.

Meng: So the savings decompose into two mechanisms. The policy avoids repeating expensive actions when the decision is already settled, and it concentrates strong verification on the few candidates that could actually flip the final answer.

Jane: Those two mechanisms attack different kinds of waste. Best-of-N over-generates, self-evaluation over-verifies, and this method does a limited amount of each, guided by cheap evidence.

Tom: Page seven then pulls the discussion together — where the savings come from, what the residual gap means, and what the field should report in future work.

Page 7 of the paper: Tom: Page seven sharpens the cost-savings story into something almost surgical. Best-of-16 spends on diversity — sixteen full generations even when answers already agree. Self-evaluation spends on evaluation — judging nearly every candidate regardless of whether it matters.

Jane: The method sits between those extremes. It buys a modest amount of diversity first, then uses cheap evidence to decide whether the next unit of compute should be another sample or a stronger verifier call. That's the allocation view made operational.

Lu: The residual gap gets a precise diagnosis, too. On eyeME 2025 and some symbolic tasks, the oracle finds correct candidates that routing misses. Those cases call for sharper uncertainty or process signals, because extra always-on verification just rescores the same ambiguous pool.

Meng: The paper then proposes a reporting norm for the whole field — action mix, cost-accuracy frontier, and oracle gap together. That makes routing claims falsifiable, because you have to show both where compute was removed and where it was concentrated.

Lalam: That norm would change how the community reads papers. Accuracy alone can hide whether a method bought better candidates, spent more verifier effort, or simply stopped earlier. The allocation trace exposes the mechanism.

Jane: The future directions are concrete as well. Natural science and code generation bring longer horizons, tool use, and public checks like unit tests. Latency could become a routed objective rather than just an audited statistic.

Tom: And the learned controller gets another chance as process signals get denser. The paper doesn't close that door, but it's clear the transparent policy wins on today's evidence.

Lu: The two failure modes from the results section come back as design guidance. Correct answer in the pool but unselected — improve uncertainty estimation or verification. No correct answer in the pool — expand generation. The oracle gap becomes a roadmap.

Meng: The conclusion ties it to what a practical system should expose: the answer, the computation path, and the marginal action that changed the decision. That connects model capability, verifier evidence, latency, and user value into one trace.

Lalam: For local deployments, where moderate sampling is feasible and deep evaluation of every candidate is not, that trace is the difference between an expensive black box and an auditable reasoning system. That's the operational payoff of the whole paper.

Tom: And that is exactly the place to close the discussion. The concluding section lands on the design implications for future test-time systems.

Conclusion: Tom: So wrapping it up — the paper reframes test-time scaling as a compute-allocation problem. The question isn't just how many samples or how long a chain of thought; it's which action deserves the next unit of compute across generation, verification, and stopping.

Jane: The evidence holds across three local generators and five reasoning benchmarks. The strongest routed variant matches the accuracy of best-of-16 majority voting and the self-evaluation weighted proxy while cutting parameter-weighted compute by 49 to 59 percent. And it does that on shared candidate pools, so the gain is genuinely allocation.

Lu: The oracle gap keeps things honest. At 91 point 36 percent pooled oracle versus 85 point 13 percent for the routed system, real headroom remains, and the paper maps that headroom into concrete next steps — sharper uncertainty, denser process signals, better candidate generation.

Meng: The allocation trace is probably the most durable contribution. When a system can say which action changed the final decision, failures become actionable instead of just scores on a leaderboard.

Lalam: The broader message is that generation and verification should be treated as one shared budget. The field gains a vocabulary — action mix, cost-accuracy frontier, oracle gap — that should make the next round of test-time scaling papers comparable in a way they haven't been.

Jane: I also want to carry the negative result on the learned controller forward. Publishing it keeps the story honest and marks a clear open problem for anyone who wants to learn a routing policy from offline trajectories.

Tom: Right, and the hand-designed policy being fully transparent means its thresholds can be audited and replayed, which is more than many approaches offer. It's a solid, reproducible contribution.

Lu: And the practical message for anyone serving reasoning models locally is simple. Don't spend the hard budget on every example; score cheaply first, then escalate only where it can change the answer.

Meng: That's a strong closing point. This paper gave us a good discussion, and we'll leave the computation path behind with it.

Jane: Looking forward to the next one. Thanks for the conversation, everyone.