Held-out test suite
The frozen scenarios behind every reported number.
The test suite was built after submission and frozen before any policy was scored on it. It is disjoint from each experiment's training and validation data. Every arm trains three seeds, and each run is scored at its terminal checkpoint with greedy decoding. Full details are in the paper, Appendix B (Frozen test suite).
Contents#
6,278 rows in total. Variants within a setting share the same scenarios, so they compare directly.
| Setting | Test groups | Rows |
|---|---|---|
| weather-regime | chickpea and potato: id, drought, wet, hot, cold | 1,280 |
| crop-variety | 4 seen + 4 held-out crops | 1,024 |
| price-regime | id, high-corn, high-soybean | 384 |
| observation-schema | S1, S2a–S2d, S3, S4, S5 | 1,024 |
| action-menu | lnpkw, lnw, ln, lw | 512 |
| reward-formulation | yield, profit, water, nutrient | 512 |
| cross-simulator | WOFOST, DSSAT, Cycles | 386 |
| unified policy | 5 seen-control + 4 held-out tuples | 1,156 |
NN baselines use _nn copies of the Tier 1 and action-menu suites: the same scenarios in numeric form.
Things to know#
- Weather-regime groups. The test groups come from a fixed temperature and rainfall rule. The
idgroup holds what is left after the four extreme groups, so it is milder than a random draw. - Cycles rebuild. The first Cycles suites reused the validation weather years. They were rebuilt from the unused years 2004–2016 (13 years, 130 rows each), so these targets rest on 13 distinct scenarios.
- Audits. Overlap checks use what determines each episode, such as crop, year and location.
- Row counts. Rows count configurations. One physical scenario can appear in several paired variants.
Get it#
The suites, with a manifest.json (config, seed, source revision, row counts), an
audit.json (overlap checks) and checksums per suite, are one Hugging Face dataset:
AgriManager-Framework/agrimanager-test-suite. The build and audit scripts
are in experiments/rebuttal_confirmatory_eval/ of the
code repository.
To rebuild a suite, run its run_build_*.sh script, then its audit_*.py. The same seeds
(20260725; 20260926 for the Cycles rebuild) reproduce the same rows. Compare them
against the published checksums.
The rebuilt suites are e4_t31_cross_sim_cycles_v2, e4_t32_cycles_profit_v2 and
e4_t32_cycles_yield_v2 (run_build_cycles_v2_eval_suites.sh). The evaluator's default
groups still name the first build, so pass them with --suite-group, as the rows of
results/scoring_manifest.tsv do.
Score against it#
bash entrypoints/eval/e5_evaluate_checkpoint.sh \ --checkpoint-repo AgriManager-Framework/schema-t21-observation-think-r2 --revision step-200 \ --suite-repo AgriManager-Framework/agrimanager-test-suite \ --results-repo local/none --tier t21 --no-upload
For the full protocol and other policies, see Evaluation.