LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration Protocol Pays OffCost-Aware Protocol Routing Across Reasoning Tasks

The one-sentence versionCentral finding

Concretely: a post-answer, pre-collaboration gpt-oss-120b probe ranks Baseline failures at 0.8847 AUROC (4,151 parseable of 4,181; 95% CI [0.8732, 0.8955]). The same score stays useful for “does any collaboration help” (0.7683 AUPRC), but is much weaker for PER-specific (0.1674 AUPRC) and Broadcast-specific (0.1041 AUPRC) value. Confidence supports initial escalation; protocol-specific cost-aware routing remains unresolved.

Plain languageWhat the paper says

Multi-agent LLM systems get better at reasoning by spending more computation. That raises a deployment question the field mostly skips: when is the extra collaboration actually worth paying for?

To answer it cleanly, we ran every problem under all four protocols and held the solver fixed within each setting. Because every problem has a realized outcome and a realized token cost under every protocol, we can score any routing policy offline against what actually happened, instead of guessing.

What we see is a pair of opposite failures. Cost-conservative policies under-escalate: they stay cheap on problems that stronger collaboration would have solved. Higher-solve frozen LLM routers over-escalate: they buy the extra solves by spending far more tokens than the problem needed. Aggregate routing accuracy hides this, because the two error types cancel in a single score.

That motivates a distinction the paper makes explicit and then measures: predicting failure risk is a different, easier problem than predicting collaboration value. A model's own post-answer confidence handles the first. It does not handle the second.

OrientationWhat is protocol routing?

A router picks one action before it sees how any protocol turns out. It looks at the problem text and the metadata it is allowed to see, and it chooses Baseline, Single, PER, Broadcast, or None — and then that choice is executed and billed. Fixed policies (“always Baseline”) are just degenerate routers.

Routers and confidence probes in this work never receive gold answers, correctness labels, oracle labels, or protocol outcomes. The None action abstains; it can be both an oracle label and a router prediction.

We score a policy on three things: solve rate, average tokens, and excess tokens — the mean positive per-problem token overpayment relative to the realized oracle. Choosing a protocol when the oracle label is None counts as over-escalation, and all of those tokens are excess.

The fixed-order oracle, and what it is not

For each problem, the oracle is the first successful protocol in the fixed aggregate cost order:

• Baseline
• →
• Single
• →
• PER
• →
• Broadcast
• →
• None

None means all four observed executions failed.

The four actionsProtocols, held on a fixed solver

In the main benchmark, every problem is run once under each of four protocols with the same gpt-oss-120b solver stack, at temperature 0.0. Changing only the collaboration protocol — never the base model — is what isolates protocol value from a change in model capability.

Token and solve figures above are on the main held-out split. Note the span: Broadcast costs roughly 34× Baseline in tokens to add 32.6 points of solve rate — averaged over problems that mostly did not need it. That gap is the entire motivation for routing.

The core distinctionFailure risk is not protocol value

These sound like the same question. They are not, and the gap between them is the paper's main result. Imagine one hard math problem and a model that has just produced a cheap Baseline answer.

The table below is the cleanest statement of the effect: one score, four increasingly specific targets. As the target narrows from “Baseline fails” toward a named protocol, ranking quality holds up moderately (AUROC stays above 0.72) but precision collapses (AUPRC falls from 0.8950 to 0.1041). The protocol-specific AUPRCs do exceed their rare prevalences — 0.1674 against 8.8%, 0.1041 against 4.2% — so confidence is not devoid of value signal. It is simply not a reliable protocol selector.

Table scrolls horizontally on narrow screens.

Source: anc/failure_and_protocol_value_targets.csv, rows for gpt-oss-120b / OmniMath. Of 4,181 matched problems, 4,151 produce parseable scores (99.28%); the 30 unparseable outputs are excluded rather than imputed.

The probe cannot cheat

After gpt-oss-120b produces its Baseline answer, we ask it for P(Baseline correct). The probe sees only the problem, the allowed metadata, and its own Baseline final answer. It sees no reasoning trace, no gold answer, no correctness label, no oracle label, and no collaboration outcome. Failure risk is then 1 − P(correct).

Main empirical resultsRouting policies on the held-out split

All policies below are evaluated on the same primary 423-problem held-out test split, so solve rates and costs are directly comparable. Intervals are 95% percentile intervals from 2,000 problem-level bootstrap resamples.

Table scrolls horizontally on narrow screens.

Source: anc/main_routing_heldout.csv. Protocol executions and router/probe calls use deterministic decoding, so these intervals measure benchmark-problem sampling uncertainty, not fresh-run variability.

How to read this table

• The self-confidence gate is the cost-efficiency result. A single rule — keep Baseline at confidence ≥ 70, otherwise choose Single, with missing confidence scores also choosing Single — reaches 78.0% solve at 45.0K tokens, versus 73.8% at 71.3K for the frozen gpt-oss-120b router. The solve intervals overlap, so this supports a cost-efficiency comparison, not a claim of higher solve rate.
• The gate is deliberately narrow. It never chooses PER, Broadcast, or None. Its 14.4-point gap to the fixed-order oracle is exactly the unsolved part.
• Errors are directional, and they point opposite ways. Tier-majority under-escalates on 27.4% and over-escalates on 12.5% of test problems. The gpt-oss-120b router cuts under-escalation to 18.0% but raises over-escalation to 33.3%. Higher-solve Llama and Gemma frozen routers reduce under-escalation to 6–11% while over-escalating on 63–71%.
• Cheap references are competitive. The selected metadata-only and text+metadata logistic routers solve 60.8% and 61.2%, both below the 65.0% of the Tier-majority heuristic.
• No evaluated router approaches the oracle operating point. Larger cross-family frozen routers reach 80–83% solve while spending 156K–280K tokens on average.

Scope of evidenceDataset and model breadth

Matched four-protocol outcomes cover 10 complete model-condition settings: two solvers, gpt-oss-120b and Gemma-4-31B-it, each across five conditions — OmniMath (n = 4,181), JEEBench (n = 515), SciBench (n = 565), LAB-Bench strict (n = 741), and LAB-Bench text-no-tool (n = 1,542). These span three broad task families: competition math, engineering entrance-exam STEM and college-level science, and biology multiple choice.

Stronger collaboration has recoverable value in all 10 paired settings: retrospective fixed-order-oracle coverage exceeds Baseline by 23.2–58.3 points. But the protocol profile varies. Broadcast is the strongest fixed protocol in nine settings, while PER exceeds Broadcast by 2.7 points for Gemma on SciBench. This supports task-dependent protocol value, not universal generalization.

The expensive protocols are not interchangeable

Conditional on Baseline and Single both failing, Broadcast exceeds PER by 9.6–10.7 points on OmniMath and 30.8–44.6 points on LAB-Bench. Nevertheless, PER-only successes remain in every setting — 2.4–11.1% of these conditional subsets — so Broadcast does not pointwise dominate PER. A router that always escalates to the most expensive protocol therefore leaves solves on the table as well as burning tokens.

Source: anc/per_broadcast_interaction.csv.

ArtifactsCode and data quickstart

The release has three pieces: the code repository, the dataset card with matched protocol traces and per-problem outcome labels, and the model card with the trained router checkpoint. Machine-readable aggregate tables ship alongside the paper; every number on this page comes from them.

To get the code and set up an environment:

git clone https://github.com/ChihHsuan-Yang/EMNLP_Cost-Aware-Protocol-Routing.git
make setup
make test

See the repository for the exact, tested commands. The lines above are the generic entry points only; the README is authoritative for environment setup, data download, and the analysis and figure pipelines. Pick a reproduction path in Reproduce the paper below.

Code

github.com/ChihHsuan-Yang/EMNLP_Cost-Aware-Protocol-Routing

Dataset card

huggingface.co/datasets/AgentsSci/EMNLP_Cost-Aware-Protocol-Routing

Model card

huggingface.co/AgentsSci/EMNLP_Cost-Aware-Protocol-Routing

Paper

arXiv:2608.14927 · PDF

Trace and per-problem outcome artifacts are subject to each upstream benchmark's terms.

Run it yourselfReproduce the paper

There are three distinct entry points, ordered by what they cost you. Start at the top; each one is a superset of the effort of the one above it.

All three assume you have done this first:

git clone https://github.com/ChihHsuan-Yang/EMNLP_Cost-Aware-Protocol-Routing.git
cd EMNLP_Cost-Aware-Protocol-Routing
make setup

make setup creates a local virtual environment and installs the package into it; the other targets use it automatically.

make smoke

make reproduce-tables

python -m protocol_routing_exec.run_protocol \
  --dataset-config  configs/datasets/fixture_tiny_math.yaml \
  --model-config    configs/models/gpt_oss_120b.yaml \
  --protocol-config configs/protocols/baseline.yaml \
  --provider-config configs/providers/mock.yaml \
  --limit 4 --output-dir outputs/mock_demo

The reproducibility boundary

Paper-to-code matrix

docs/reproduction/paper_artifact_matrix.md

Dataset schema

Dataset card — fields, splits and outcome labels

Model card

Model card and router checkpoint on Hugging Face

Repository

github.com/ChihHsuan-Yang/EMNLP_Cost-Aware-Protocol-Routing

Honest scopeLimitations

These are the paper's own stated limitations, and they bound how far every number on this page travels.

• Narrow main comparison. The main router comparison and the pre-answer confidence gate use one solver family and one math benchmark.
• Targeted, not universal, breadth. The broader paired matched-outcome study covers four benchmarks across three broad task families and two solvers. Post-answer confidence and held-out router analyses cover the narrower six settings formed by both solvers on OmniMath and the two LAB-Bench conditions. These are targeted robustness checks, not a claim of broad universality.
• One realization per cell. Each problem–protocol pair has one deterministic realized outcome. The 2,000-resample intervals quantify variation over benchmark problems, not fresh-run stochastic variability.
• The oracle is a diagnostic, not a policy. The fixed-order oracle is retrospective and uses an aggregate cost order; it is neither deployable nor an estimate of per-instance expected utility.
• Cost is measured in tokens. We measure cost primarily in logged tokens. Latency, monetary price, energy, parallelism, and answer quality beyond exact correctness can change deployment preferences.
• Probe exclusions and variance. The post-answer probe excludes 30 unparseable outputs from its probability metrics, and confidence quality varies substantially by model and domain.
• No causal mechanism. The observed PER–Broadcast differences do not identify a causal mechanism; understanding why protocol value changes by task remains open.

CitationCite this work

Please cite the arXiv preprint. The EMNLP 2026 proceedings entry does not exist yet; we will update this block when volume and page numbers are assigned.

@article{yang2026protocolrouting,
  title  = {LLMs Can Predict Failure Risk, But Struggle to Predict Which
            Collaboration Protocol Pays Off: Cost-Aware Protocol Routing
            Across Reasoning Tasks},
  author = {Yang, Chih-Hsuan and Jiang, Jingyan and Yang, Cheng-Hau and
            Vasudevan, Vikram and Zheng, Huihuo and Vishwanath, Venkatram
            and Thakur, Rajeev},
  year   = {2026},
  eprint = {2608.14927},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url    = {https://arxiv.org/abs/2608.14927},
  note   = {Accepted at EMNLP 2026}
}

SupportAcknowledgments

This research used resources of the Argonne Leadership Computing Facility, a U.S. Department of Energy (DOE) Office of Science user facility at Argonne National Laboratory (ANL) operated under Contract No. DE-AC02-06CH11357.

AuthorsTeam and contact

• Chih-Hsuan YangArgonne National Laboratory
• Jingyan JiangArgonne National Laboratory
• Cheng-Hau YangArgonne National Laboratory
• Vikram VasudevanOregon State University
• Huihuo ZhengArgonne National Laboratory
• Venkatram VishwanathArgonne National Laboratory
• Rajeev ThakurArgonne National Laboratory

Questions, corrections, or requests about the released artifacts: bellayang@anl.gov.