Wrong but Useful: Trajectory Value Beyond Answer Correctness in Multi-Agent Messages

The distinctionCorrectness is not value

Multi-agent systems routinely decide which messages should shape a final answer using agreement, confidence, or automated scores. That filtering assumes a message likely to be correct is also worth keeping.

This paper separates two different judgments about the same message:

• Proposal correctness — is the message's own answer right?
• Trajectory value — does making the whole message available help or harm the reasoning that follows?

These come apart in both directions. A wrong answer can supply a useful decomposition or constraint; a correct answer can be paired with misleading reasoning that disrupts integration. The paper's conclusion: answer correctness is informative, but it does not determine trajectory value.

MethodDiverse Hypothesis Deliberation

DHD is a controlled measurement protocol. For each problem a recruiter assigns five problem-specific roles without seeing the ground-truth answer. Five hypothesizers each produce one structured message. An integrator then synthesizes a final answer, and a separate evaluator compares that answer to ground truth.

The measurement comes from replay: the five messages are cached, and the same integrator is run again with each message either available or hidden. Comparing the outcomes isolates one message's contribution — its trajectory value — while holding everything else fixed.

Decoding is fixed per role: recruiter, integrator and evaluator run at temperature 0; hypothesizers run at temperature 0.7 so their solution paths differ. Both model families are scored by the same separate gpt-oss-120b evaluator.

ResultsWhat the study found

Across five mathematics and science benchmarks and two openly available model families — gpt-oss-120b (OSS) and gemma-4-31B-it (Gemma) — wrong-helpful messages appear in every benchmark–model combination.

Among flips caused by wrong-answer messages, more than four in ten are helpful

Restricted to in-pool leave-one-out replays that actually change final correctness, the helpful share for messages with a wrong proposed answer is 41.9% for OSS and 45.3% for Gemma. A problem-cluster bootstrap places the OSS share at 39.5–44.3%.

The effects survive controlled repetition

Repeated replay shows the number of repeatable message effects is unlikely to arise from replay variation alone (p = 0.0002 against a within-block permutation control).

The complete message works best — but why is open

A focused intervention on repeatable wrong-helpful messages found that the complete message works best, and that retaining its reasoning preserves more success than retaining only its answer. The paper states plainly that the source of the complete-message advantage remains open.

Evaluation is stable across evaluators

16,724 saved submissions replayed with three open evaluators. Meta-Llama-3.1-70B-Instruct and gemma-3-27b-it appear here only as evaluators in this agreement check — they are not actor model families in the study.

Benchmark sizes and per-benchmark terms are in Table 3.

† MaScQA contains 650 raw rows; one row is an exact duplicate, so all analyses collapse it to 649 unique problems. Some released per-benchmark property files therefore list 650 raw rows while the analysis N is 649.

ResourcesEverything released, in one table

Each row links directly to the artifact. The dataset is public but not unrestricted — see the terms column before you download.

Benchmarks and model families

^† The dataset registry records Apache-2.0 for Omni-MATH-2 while the authors’ own DATA.md disagrees; we have not adjudicated this, so treat those terms as unsettled. ^‡ MaScQA contains 650 raw rows; one is an exact duplicate, so all analyses collapse it to 649 unique problems.

Actor and evaluator models

Llama-3.1-70B and gemma-3-27b-it appear only in the evaluator-agreement check. They are not actor model families in this study.

ReproducibilityReproduce the study

There is an important distinction between offline analysis reproduction and live model regeneration, and they are not equally available.

Offline reproduction, from a clean machine

git clone https://github.com/ChihHsuan-Yang/AAAI_Wrong-but-Useful.git
cd AAAI_Wrong-but-Useful
python3 -m venv .venv && source .venv/bin/activate   # Python >= 3.11
python -m pip install -r requirements.lock
python -m pytest                    # unit tests
python scripts/validate_inputs.py   # checksum-validate bundled data
bash scripts/reproduce_analysis.sh  # regenerate figures and summary tables

No API key, endpoint, cluster account, or author filesystem is needed for this path. To run new model calls you supply your own endpoint through environment variables; a dry run prints the resolved per-role model identities before any request is made.

Full details, expected outputs, troubleshooting and a per-artifact reproduction map are in the repository README, docs/PAPER_REPRODUCTION_MAP.md and docs/MODEL_DEPENDENCIES.md. The paper's arXiv entry also carries an ancillary reproducibility artifact.

AccessData and models

No model is released, and none is needed

Dataset access

The full measurement archive is public at AgentsSci/AAAI_Wrong-but-Useful. It contains third-party benchmark question text and raw model messages, so it carries the upstream benchmarks' own terms and is not unrestricted:

• Non-commercial as a whole. MaScQA content is CC BY-NC-SA 4.0, so the dataset as a whole cannot be used commercially. Excluding benchmark_id = mascqa gives a commercially usable subset.
• Do not train on the LAB-Bench subset. LAB-Bench is CC BY-SA 4.0 and carries an upstream do-not-train request. The canary string that would normally enforce this automatically is absent from the released files (measured: 0 of 1,542 rows), so a contamination filter will not catch it — exclude benchmark_id = labbench explicitly.

Read the dataset card before downloading. Offline reproduction does not require the dataset: the derived data and figure inputs for that path are bundled in the code repository.

Offline reproduction does not depend on it: the derived data and figure inputs needed for the Level 1 path are bundled in the code repository itself.

Honest scopeLimitations

Quoted from the paper, because they qualify every result above.

• Trajectory-value labels describe a message–pool–integrator context, not an intrinsic property of text; individual signs can change even with fixed problems and messages.
• Repetition reduces but does not eliminate uncertainty, and the fifth-block analysis measures same-problem opportunity rather than unseen-problem generalization.
• Leave-one-out hides a whole message in one fixed prompt order; the smaller masking diagnostic only begins to separate reasoning from the answer field.
• Nested K prefixes are not randomized agent additions, and one model family fills all reasoning roles within each run.
• Results may not transfer to interactive debate, heterogeneous agent models, frontier models, or tasks without a stable ground-truth answer.
• Gemma rates condition on complete replay records, and compound labels are more evaluator-sensitive than individual answers.

CitationCite this work

@misc{yang2026wrongbutuseful,
  title         = {Wrong but Useful: Trajectory Value Beyond Answer
                   Correctness in Multi-Agent Messages},
  author        = {Yang, Chih-Hsuan and Chowdhury, Anjir Ahmed and
                   Yang, Cheng-Hau and Zheng, Weijian and
                   Llorente, Fernando and Ma, Xiaolong and Li, Xinyang and
                   Huerta, Eliu A. and Foster, Ian T. and Thakur, Rajeev},
  year          = {2026},
  eprint        = {2608.14375},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2608.14375}
}

Acknowledgments

This research used resources of the Argonne Leadership Computing Facility, a U.S. Department of Energy (DOE) Office of Science user facility at Argonne National Laboratory (ANL) operated under Contract No. DE-AC02-06CH11357. The work was also supported under the same contract by the DOE Office of Science's Advanced Scientific Computing Research Program and by Laboratory Directed Research and Development (LDRD) funding from ANL, provided by the Director, DOE Office of Science.