AgriManager / Docs
Website GitHub

Held-out test suite

The frozen scenarios behind every reported number.

The test suite was built after submission and frozen before any policy was scored on it. It is disjoint from each experiment's training and validation data. Every arm trains three seeds, and each run is scored at its terminal checkpoint with greedy decoding. Full details are in the paper, Appendix B (Frozen test suite).

Contents#

6,278 rows in total. Variants within a setting share the same scenarios, so they compare directly.

SettingTest groupsRows
weather-regimechickpea and potato: id, drought, wet, hot, cold1,280
crop-variety4 seen + 4 held-out crops1,024
price-regimeid, high-corn, high-soybean384
observation-schemaS1, S2a–S2d, S3, S4, S51,024
action-menulnpkw, lnw, ln, lw512
reward-formulationyield, profit, water, nutrient512
cross-simulatorWOFOST, DSSAT, Cycles386
unified policy5 seen-control + 4 held-out tuples1,156

NN baselines use _nn copies of the Tier 1 and action-menu suites: the same scenarios in numeric form.

Things to know#

Get it#

The suites, with a manifest.json (config, seed, source revision, row counts), an audit.json (overlap checks) and checksums per suite, are one Hugging Face dataset: AgriManager-Framework/agrimanager-test-suite. The build and audit scripts are in experiments/rebuttal_confirmatory_eval/ of the code repository. To rebuild a suite, run its run_build_*.sh script, then its audit_*.py. The same seeds (20260725; 20260926 for the Cycles rebuild) reproduce the same rows. Compare them against the published checksums.

Rebuilt Cycles suites

The rebuilt suites are e4_t31_cross_sim_cycles_v2, e4_t32_cycles_profit_v2 and e4_t32_cycles_yield_v2 (run_build_cycles_v2_eval_suites.sh). The evaluator's default groups still name the first build, so pass them with --suite-group, as the rows of results/scoring_manifest.tsv do.

Score against it#

bash
bash entrypoints/eval/e5_evaluate_checkpoint.sh \
  --checkpoint-repo AgriManager-Framework/schema-t21-observation-think-r2 --revision step-200 \
  --suite-repo AgriManager-Framework/agrimanager-test-suite \
  --results-repo local/none --tier t21 --no-upload

For the full protocol and other policies, see Evaluation.