SciAgentTrace

What SciAgentTrace is

A workload-characterization resource for scientific agents

SciAgentTrace records the same scientific problems being solved repeatedly under different agent protocols, and keeps the operational evidence of each run. It is built to support ML workload characterization and data-centric AI research: the unit of study is the request structure a protocol generates, not just the answer it returns.

The corpus is matched by construction. A single fixed set of 6,653 problems is executed under two actor models and four protocols, producing 53,224 trajectories arranged in 40 complete actor-family-protocol cells. Because every cell sees an identical problem set, differences between protocols can be read without re-weighting for problem difficulty or benchmark mix.

• matched panel

  One problem set, every condition

  6,653 problems × 2 actor models × 4 protocols = 53,224 trajectories, with no cell left partially populated.

• execution layer

  Operational evidence, not just outcomes

  Client-observed elapsed time on every matched trajectory; token totals and observable model-request counts wherever they can be attributed.

• provenance

  An availability mask, not imputed zeros

  Fields that cannot be attributed to a single request stay unavailable and are marked as such. Nothing is filled in with a zero.

The full package is broader than the matched core panel: it retains runs outside the strict 6,653-problem matched design. Analyses that depend on matching should use the core panel.

Why it is different

The execution layer

Most agent-protocol datasets record what was said and whether the final answer was correct. SciAgentTrace additionally records what the run cost the serving system, on the same matched design — which is what makes protocol comparison a workload question rather than only an accuracy question.

Client-observed elapsed time

Wall-clock duration as observed at the client, recorded for all 53,224 matched trajectories. This is the one execution-layer field with complete coverage across the matched panel.

Successful-response token totals

Token accounting for responses that completed successfully, available on 39,918 matched trajectories.

Observable model-request counts

The number of model requests visible to the client, available on the same 39,918 matched trajectories. Request count is the field that carries most of the structural difference between protocols.

Dataset composition

Five benchmark families, five scientific domains

The matched problem set is drawn from five established benchmark families spanning competition mathematics through experimental biology and materials science. Each family is carried through all eight model × protocol conditions.

¹ The per-family trajectory column is derived as problems × 2 actor models × 4 protocols; it sums to the verified panel total of 53,224. ↩

Actor models

• actor arm 1

  Gemma 4 31B

  One of two actor models executing every protocol across every benchmark family.

• actor arm 2 · evaluator

  GPT-OSS 120B

  The second actor arm, and also the fixed answer evaluator for both arms — so outcome labelling is held constant across the comparison.

Protocols

Four execution protocols

Each protocol is a different way of spending compute on the same problem. They range from a single request to a multi-agent exchange, which is what produces the spread in request structure that the execution layer captures.

• protocol 1

  Direct Baseline

  A single pass at the problem with no iteration and no additional roles. This is the reference condition against which the other three are compared.

• protocol 2

  Single-Agent Iterative

  One agent revisits and revises its own work across multiple turns, adding requests without adding roles.

• protocol 3

  Planner-Executor-Reviewer (PER)

  Work is split across three distinct roles — planning, execution and review — in a structured pipeline.

• protocol 4

  Broadcast Deliberation

  Multiple agents exchange messages over a shared broadcast channel before an answer is settled.

Measurement coverage

An audited coverage boundary

Execution-layer coverage is not uniform, and the corpus states exactly where it stops. Elapsed time is complete across the matched panel. Token totals and request counts are available on 39,918 of 53,224 trajectories.

Workload findings

What the execution layer shows

Two headline results from the corpus. Both are descriptive characterizations of the released workload, not controlled serving-system experiments.

Protocols spend very different amounts of wall-clock time

Measured as equal-family median elapsed ratios against the Direct Baseline, the other protocols range from 1.06× to 6.18× under Gemma 4 31B and 1.55× to 8.38× under GPT-OSS 120B. The spread is wide enough that protocol choice, not model choice alone, dominates the time a workload takes.

Request structure carries information that tokens alone do not

In held-out-problem prediction of elapsed time, adding request count and protocol identity to a tokens-only predictor raises R² from 0.695 to 0.789 for Gemma 4 31B and from 0.773 to 0.803 for GPT-OSS 120B. Token volume is not a sufficient statistic for the cost of an agent workload; how the work is divided into requests matters on its own.

Access status

Availability

Where to get it

Dataset

huggingface.co/datasets/AgentsSci/IEEE2026_BigData_MAS-4-Science-Matching — trajectories, outcomes, matched outcomes, problems, ordered trace events, execution records, registry, and integrity manifests.

Code

github.com/ChihHsuan-Yang/IEEE2026_BigData_MAS-4-Science-Matching — the protocol runtime for all four protocols, the trace-processing pipeline, benchmark preparation, and evaluation tooling.

Paper

A Data Descriptor is in preparation for Nature Portfolio Scientific Data. Nothing has been submitted to any venue and no DOI exists. A link will be added if and when a record is published.

Load it in one line

from datasets import load_dataset

repo = "AgentsSci/IEEE2026_BigData_MAS-4-Science-Matching"
traj = load_dataset(repo, "trajectories", split="test")

Using it correctly: match problem_uid across the protocols you compare, name the actor and evaluator, and match the execution configuration before comparing elapsed time. Preserve unavailable fields rather than coercing them to zero.

Licensing

Composite terms — there is no single license

SciAgentTrace extends upstream benchmark families that carry different and partly incompatible terms. The corpus therefore cannot be offered under one permissive license, and reuse obligations differ by constituent family.

Because the non-commercial and do-not-train conditions attach to specific subsets, users combining families must satisfy the strictest terms in the combination they actually use.

Citation

How to cite

The manuscript is in preparation and has not been submitted; no DOI exists. The entry below is a

@misc{sciagenttrace2026,
  title        = {SciAgentTrace: A Matched Cross-Domain Corpus of
                  Scientific-Agent Protocol Executions},
  author       = {Yang, Chih-Hsuan and Thakur, Rajeev},
  year         = {2026},
  note         = {Data Descriptor in preparation for Scientific Data. Not submitted; no DOI assigned.},
  howpublished = {\url{https://huggingface.co/datasets/AgentsSci/IEEE2026_BigData_MAS-4-Science-Matching}},
}

Links

Resources

• Dataset huggingface.co/datasets/AgentsSci/IEEE2026_BigData_MAS-4-Science-Matching
• Code github.com/ChihHsuan-Yang/IEEE2026_BigData_MAS-4-Science-Matching
• Paper In preparation for Scientific Data (Nature Portfolio); not submitted.