Precise but Uncoupled: Reviewer Precision Does Not Guarantee Critique Uptake in Multi-Agent Math Reasoning

00What we found

A reviewer that is good at spotting wrong answers does not, by itself, make a multi-agent system solve more problems. Detection and repair come apart.

• The better detector is the worse fixer. On 4,181 Omni-MATH problems the planner–executor–reviewer pipeline (PER) has the more precise reviewer — 0.861 against broadcast's 0.644 — yet reaches lower final accuracy, 85.2% against 89.2%.
• The gap is in uptake, not in detection. Evaluator-verified useful critique changes the candidate PER carries forward only 33.6% of the time, against 93.5% under broadcast. Realized repair is 5.6× lower: 0.051 against 0.286.
• Detection and repair are separable, and it generalises. Across five benchmarks and two actor families, PER's reviewer precision beats broadcast's in 10/10 cells, while inside PER precision exceeds verified repair in 9/10 (the exception has only two evaluable follow-ups) and inside broadcast in 10/10.
• Why it matters. Reviewer-centric evaluation can overstate system quality. A protocol may spot errors well and still fail to solve more problems, so a review stage must be judged on what the system does next, not only on whether the critique was right.

The three numbers

Paper Table 11, review-conditioned decomposition. N = 4,181; 10,804 PER and 11,587 broadcast review instances.

01The result in two figures

The first figure is the mechanism. The second is the outcome it produces.

Both figures regenerate from the released data. Method, the full tables and the post-rebuttal robustness matrix follow below.

02Abstract

Many math- and science-oriented agent systems use hierarchical designs with specialized reviewer roles, assuming that a dedicated review stage should help turn wrong candidates into correct ones. We test this assumption on 4,181 verifier-grounded Omni-MATH problems using matched gpt-oss-120b actors. Collaboration adds little on the easiest tiers, but from tier 4 onward the gains open sharply; in this harder regime, broadcast-style peer discussion reaches higher final accuracy than a planner-executor-reviewer pipeline (PER). We ask whether this gap is explained by reviewer quality or by whether critique changes the next answer the protocol carries forward. It is not explained by reviewer precision alone: PER's reviewer is more precise than broadcast's (0.861 vs. 0.644), yet evaluator-verified useful critique is much less likely to change the next candidate and produces lower reviewer-guided repair. These results show that reviewer detection quality and critique uptake are empirically separable. Within matched PER interventions, forcing explicit acknowledgment lowers final accuracy, while embedding reviewer guidance directly in the solver's working context partially improves follow-through without closing the gap. Overall, reviewer-centric evaluation can overstate system quality: a protocol may spot errors well yet still fail to solve more problems if it does not act on those critiques.

Abstract as published in the arXiv v1 metadata record. The PDF's own abstract is slightly longer; where they differ, the downloadable artifact is arXiv v1.

03Why precise review need not produce repair

Many math- and science-oriented agent systems add a dedicated reviewer role, on the assumption that routing critique through a review stage turns wrong candidates into correct ones. That assumption bundles two separate things together.

A reviewer role has to do two jobs, and they can be measured apart:

1. Detection — does a reviewer warning actually point at a real error? That is reviewer precision.
2. Uptake and repair — once critique is verified useful, does the protocol change the next candidate it carries forward, and does that change fix the answer?

Reviewer-centric evaluation measures the first and treats the second as implied. This paper measures both and finds they come apart: the protocol with the better reviewer is the one that acts on critique less. The reason is structural rather than a property of the critique text — it is about where the critique lands and who has to agree before an answer is submitted.

The structural contrast

Two protocols differ in how critique reaches the answer. The rows below are the paper's Figure 1, re-rendered here as text.

Re-rendering of Figure 1, Why reviewer quality and realized repair can diverge. In PER, information flow and decision flow can separate: a reviewer can send a correct warning without changing the next candidate. In broadcast, critique is embedded in shared candidate state and group approval, making bypass harder by design.

05Main findings

Primary result: precision and uptake come apart

Paper Table 11, review-conditioned decomposition. N = 4,181; 10,804 PER and 11,587 broadcast review instances.

→ scroll the table sideways

Where the accuracy difference lives

Collaboration is worth little on the easiest problems and a great deal on the hardest ones. Over N = 4,181 the four rungs reach FinalPassRate 56.8% (baseline), 78.8% (single-agent iterative), 85.2% (PER) and 89.2% (broadcast); Pass@1 is 56.8 / 57.4 / 72.8 / 78.6%.

Within-PER interventions: directional, not causal

Two probes change only how critique reaches the solver, holding model family, reviewer setup and outer-loop budget fixed. Forcing the solver to acknowledge critique explicitly (ACK-required) lowers final accuracy, from 85.2% to 82.5%, and roughly doubles NeglectRate (0.488 → 0.792). Embedding reviewer guidance directly in the solver's working context (EMB) partially recovers, to 86.3%, without closing the gap to broadcast's 89.2%.

→ scroll the table sideways

Robustness inside the paper's own setting

• The coupling metric is definition-stable. Legacy and strict-deterministic CouplingRate are numerically identical for both protocols (PER 0.336 = 1,766/5,253; broadcast 0.935 = 4,876/5,214), with strict-vs-legacy disagreement of 0/5,253 and 0/5,214. An equivalence-aware variant moves PER only to 0.340. (Paper Table 3.)
• The result is not a single-evaluator artifact. Replaying 16,724 saved submissions under three evaluators — gpt-oss-120b, Meta-Llama-3.1-70B-Instruct, gemma-3-27b-it — gives instance-level disagreement of 3.38–5.76% and Cohen's κ 0.850–0.915. Re-tabulating those labels preserves the broadcast−PER ordering: +2.88 pp [1.82, 3.91] under GPT-OSS, +3.21 [2.13, 4.26] under Llama, +3.33 [2.22, 4.45] under Gemma-3.
• But evaluator sensitivity is real where it matters most. On the hard collaborative slice (PER and broadcast, tiers 7–10, N = 1,694) disagreement roughly doubles, to 9.68% / 9.39% / 5.25% with κ 0.755 / 0.761 / 0.875.
• Run-to-run variation is smaller than the effect. On a frozen N = 195 subset, independent re-executions move PER FinalPass by about 0.5 pp and broadcast by about 2 pp, against a paired broadcast−PER gap of +4.10 to +6.67 pp. Direction reproduced on both tested backends; one of four intervals crosses zero.

Post-rebuttal robustness: five benchmarks, two actor families

Across the ten dataset × actor cells:

• PER reviewer decision precision exceeds broadcast precision in 10/10 cells.
• Within PER, precision exceeds verified strict next-answer repair in 9/10 cells. The exception is JEEBench × Gemma-4, where only two follow-ups are evaluable — no stable conclusion is possible from that cell.
• Within broadcast, precision exceeds verified repair in 10/10 cells.

So the separation between detecting an error and repairing it is broad across models and domains. That is the claim, and it is the only claim this matrix supports.

06What an uncoupled episode looks like

The three-step decomposition is not an abstraction over the data — it is read off individual answer transitions. Below are real transitions from the released table, all drawn from Omni-MATH (Apache-2.0) so that nothing here is encumbered by the NonCommercial or ShareAlike terms above.

Each row is one review episode where the reviewer asked for a revision and the candidate in hand was in fact wrong — a true-positive detection. All six are drawn from the strict pre-gate population defined below, so every "after" answer is the actor's own immediate response to the critique rather than a system-selected candidate. What varies is what happened next.

→ scroll the table sideways

Counting these transitions requires care, because the two protocols are not eligible on the same terms. Over the whole benchmark there are 1,992 such useful-review episodes under PER and 1,742 under broadcast. Only a subset of each has an answer pair that parses on both sides and records the actor's own immediate response rather than a system-selected candidate: the strict pre-gate population, n = 379 for PER and n = 533 for broadcast. On that population, and on that population for both protocols:

→ scroll the table sideways

The remaining changed rows in each column are transitions the evaluator did not label on both sides, so they support no repair verdict either way.

09Citation

Please cite the arXiv preprint.

@misc{yang2026preciseuncoupled,
  title         = {Precise but Uncoupled: Reviewer Precision Does Not Guarantee
                   Critique Uptake in Multi-Agent Math Reasoning},
  author        = {Yang, Chih-Hsuan and Jiang, Jingyan and Vasudevan, Vikram and
                   Yang, Cheng-Hau and Zheng, Huihuo and Chen, Le and
                   Huerta, Eliu A. and Vishwanath, Venkatram and
                   Foster, Ian T. and Thakur, Rajeev},
  year          = {2026},
  eprint        = {2607.15388},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  doi           = {10.48550/arXiv.2607.15388},
  url           = {https://arxiv.org/abs/2607.15388}
}

arXiv

arXiv:2607.15388 [cs.AI], v1, submitted 16 July 2026

DOI

10.48550/arXiv.2607.15388

Status

Accepted to NeurIPS 2026, Main Track. The arXiv record is still v1; an updated version is pending.

Contact

bellayang@anl.gov (corresponding author)