When Self-Consistency Backfires: Majority Vote Hurts the Majority of Hard Science Problems for Small LLMs

Venue: Accepted at the COLM 2026 Workshop on Efficient Reasoning

Core Finding: On the full GPQA Diamond benchmark (198 graduate-level science questions), majority voting via self-consistency reduces per-problem accuracy on a majority of problems for two instruction-tuned models from different families: 56.6% of problems for Qwen2.5-7B and 65.7% for Llama-3-8B, with Qwen the primary demonstration and Llama corroborating the direction from a near-chance baseline.

Methodology: The study uses a pre-registered confirmatory design: 47 problems are exploratory (the hypotheses below were generated from them), and the remaining 151 are a confirmatory test set. The hypotheses and thresholds were locked and git-tagged before any confirmatory analysis. All four pre-registered hypotheses passed on the confirmatory set. Models tested were Qwen2.5-7B-Instruct-Turbo and Meta-Llama-3-8B-Instruct-Lite, both small (7–8B) and non-reasoning, with N = 64 samples per problem at temperature 0.7.

Key Results:

• Backfire rate (mv gain = MV acc(64) − MV acc(1) < 0) is 56.6% (95% CI [49.5, 63.6]) for Qwen and 65.7% ([59.1, 71.7]) for Llama on the pooled 198 problems.

• The aggregate and per-problem pictures diverge. Voting barely moves aggregate accuracy (Qwen 0.342 to 0.369, Llama 0.273 to 0.313) while harming the majority of problems individually.

• The worst single problem loses 47 points (Qwen) or 46 (Llama); a few gain as much as 66 or 70.

• Chemistry shows the highest backfire rate for both models (66.7% Qwen, 68.8% Llama), physics is intermediate (50.0% Qwen, 67.4% Llama), and biology is least affected (36.8% Qwen, 42.1% Llama).

Oracle Upper Bound: "A grid oracle that routes each problem to the best majority-vote accuracy across all N in 1, 2, 4, 8, 16, 32, 64 reaches 0.482 (Qwen) and 0.439 (Llama), a ceiling 14 points above N = 1 for Qwen and 17 points above for Llama." This is a theoretical upper bound requiring ground truth, not a deployable method.

Gate Failures: Two verifier-free gates were tested and both failed:

• The agreement gate (k = 8, τ = 0.75) reaches 0.368 accuracy for Qwen and 0.312 for Llama, against 0.369 and 0.313 for fixed-budget voting at N = 64 — differences of 0.0006 and 0.0014. What the gate buys relative to a flat N = 64 is therefore compute rather than accuracy.

• The entropy gate captures 0.2% of grid-oracle headroom for Qwen and 31.2% for Llama (evaluated on confirmatory 151 only). For Qwen, the gate matches single-sample accuracy to three decimals, so it does nothing rather than actively hurting.

• Across all 22 operating points swept (k ∈ 4, 8 and τ from 0.50 to 1.00 in steps of 0.05), none beats fixed-budget meaningfully.

Mechanism: Confidence does not track correctness on these problems. Binning problems by plurality agreement: Qwen's plurality is correct 52.5% of the time in the highest-agreement bin, and Llama's only 28.6%, lower than its own low-agreement bin. Llama's accuracy is not monotone in agreement — higher agreement does not indicate higher accuracy for it.

Positioning: The paper distinguishes itself from prior work: Prior work established that self-consistency helps on benchmarks where models have higher baseline accuracy (Wang et al., 2023); we show it hurts the majority of problems on a hard one. It also differs from Chen et al. (2024) by measuring the fraction of individual problems harmed rather than the shape of the aggregate curve and by showing that on a uniformly hard benchmark, the verifier-free signals such estimation would rely on do not recover the headroom.

Limitations: Llama is near chance (single-sample accuracy 0.273 vs 0.25 random baseline); all results are on one benchmark (GPQA Diamond); only two small, non-reasoning models were tested; the original 47-problem subset was easier than the full benchmark; Monte Carlo boundary sensitivity exists (14 of Qwen's 151 confirmatory problems have exactly zero gain, and backfire is defined strictly as mv gain < 0, so the rate can only rise); the entropy threshold was selected in-sample, making its reported capture optimistic; only two verifier-free signals were evaluated.

Conclusion: "On hard reasoning problems, self-consistency backfires on the majority of problems, pre-registered and confirmed on the full GPQA Diamond benchmark for Qwen and corroborated by a second family from a near-chance baseline. The accuracy this leaves on the table is real but unreachable with a verifier-free agreement or entropy gate, because confidence does not track correctness. Whether reasoning-native models escape this is the key open question."

Improvements to AI Systems:

1. Adaptive sampling controller: Instead of fixed N=64 samples, the system dynamically stops sampling when agreement confidence plateaus, and detects when agreement is negatively correlated with correctness (as seen with Llama) to switch strategies or abstain.

2. Verifier-aware routing: The system uses a learned verifier (not agreement/entropy heuristics) to decide per-problem whether to (a) return a single sample, (b) majority vote over k samples, or (c) flag for human review—since verifier-free gates fail to recover the 14–17 point oracle headroom.

3. Per-domain calibration: The system applies domain-specific voting policies (e.g., reduce voting in chemistry/physics where backfire rates are 66–69%, increase voting in biology where backfire is only 37–42%) based on the model’s known per-subject failure modes.

4. Confidence-correctness decorrelation detector: The system monitors whether its internal confidence scores track accuracy on hard problems; if not (as with Qwen at 52.5% accuracy in the highest-agreement bin), it automatically downgrades confidence-based early stopping and switches to compute-budgeted alternatives.

5. Backfire-aware answer selection: The system compares single-sample vs. voted outputs on a per-problem basis and, when voting would reduce accuracy (detectable via internal consistency metrics on held-out validation sets), it defaults to the single-sample answer, preventing the 47-point worst-case losses.

What the improved system can do:

• Achieve higher per-problem accuracy on hard science benchmarks by avoiding majority-vote harm on 56–66% of problems.

• Recover up to 14–17 points of accuracy headroom (oracle ceiling) using verifier-based routing instead of naive voting.

• Save compute by skipping unnecessary sampling on problems where voting is known to backfire (e.g., chemistry).

• Provide reliable abstention or human-escalation flags for problems where confidence is decorrelated from correctness.

• Generalize to other hard reasoning domains by detecting when self-consistency is counterproductive and adapting accordingly.

Self-consistency (SC) via majority vote is a widely used way to spend inference-time compute: sample N chains of thought, return the plurality answer. On the full GPQA Diamond benchmark (198 graduate-level science questions), majority voting reduces per-problem accuracy on a majority of problems for two instruction-tuned models from different families: 56.6% of problems for Qwen2.5-7B and 65.7% for Llama-3-8B, with Qwen the primary demonstration and Llama corroborating the direction from a near-chance baseline. The effect was pre-registered on a 151-problem confirmatory split after being observed on 47 exploratory problems, and all four confirmatory hypotheses passed. A grid oracle that routes each problem to the best N across 1, 2, 4, 8, 16, 32, 64 marks a theoretical upper bound 14 accuracy points above N = 1 for Qwen and 17 for Llama, an oracle bound requiring ground truth rather than a deployable method. No verifier-free gate reaches it: neither a plurality-agreement gate nor a token-entropy gate moves accuracy more than 0.002 from fixed-budget voting at N = 64. The mechanism is direct: confidence does not track correctness on these problems. In the highest-agreement bin the plurality answer is correct about half the time for Qwen, and for Llama that bin is less accurate than its lowest-agreement bin. We pre-register and confirm these findings on small instruction-tuned models; we do not test reasoning-native models, which we flag as the central open question.