Evaluation
Score a checkpoint, your own model, a scripted reference or an NN baseline on the held-out test suite.
Protocol#
Every number in the paper follows these rules. Follow them to get comparable numbers.
| Rule | Detail |
|---|---|
| Frozen test suite | score on the held-out test suite; no training, checkpoint selection, prompt tuning or decoding change uses its scenarios |
| Terminal checkpoint | score the last checkpoint of each run (the stop step for a run that stopped early), with no selection on validation |
| Training response format | a think-trained checkpoint is scored in think format and a no-think one in no-think format; the evaluator reads the format from the checkpoint and asserts it |
| Greedy decoding | runtime.temperature=0, one pass (runtime.num_trials=1) |
| No turn cap | leave runtime.max_turns unset for any reported run |
| Three seeds | report mean ± SD over three training seeds per arm, and differences between arms in propagated SD |
| Raw and references | report the raw return with both reference returns and the gap; normalize only where the gap is usable (Benchmark) |
Score a released checkpoint#
entrypoints/eval/e5_evaluate_checkpoint.sh scores one checkpoint revision on every suite group of a
setting and writes one result bundle. It needs one CUDA GPU and install.sh --gpu.
bash entrypoints/eval/e5_evaluate_checkpoint.sh \ --checkpoint-repo AgriManager-Framework/schema-t32-qwen-think-r2 \ --revision step-800 \ --suite-repo AgriManager-Framework/agrimanager-test-suite \ --results-repo <your-user>/<results-dataset> \ --tier t32 \ --no-upload
| Argument | Meaning |
|---|---|
--checkpoint-repo, --revision | Hugging Face model repository and branch step-N; the terminal revision is the highest N (Models and data) |
--checkpoint-dir | score weights already on disk (used for the untrained model) |
--suite-repo | test-suite dataset repository (required); with --suite-dir it only names the suite in the bundle |
--suite-dir | score suites already on disk (a directory holding the suite-group folders) |
--results-repo | results dataset for the upload (required, also with --no-upload, where it is unused) |
--tier | t11, t11_chickpea, t12, t13, t21, t22, t23, t31, t32 |
--suite-group | override the setting's groups (repeatable), for example to score a cross-simulator specialist on the unified-policy suites |
--require-think, --think-tag | set the response format by hand, for checkpoints whose manifest does not record it |
--max-samples N | score the first N rows of each split, for a pipeline check |
--no-upload | keep the bundle local, in <work-dir>/bundle (--work-dir, default ./.e5_work); without it, HF_TOKEN is required |
The bundle holds per-scenario outputs, the metrics of every split, any failures, the applied environment
overrides, and eval_manifest.json, which ties every number to the checkpoint and suite revisions
that produced it.
WOFOST suites read weather from a public weather pool, and the evaluators download the pool the first time a
suite needs it: the maize pool (0.25 GB) for Tier 2 and the WOFOST parts of Tier 3, the 20-crop pool (17.6 GB)
for weather-regime and crop-variety. The rebuilt Cycles suites are scored with
--suite-group; see Held-out test suite.
results/scoring_manifest.tsv lists every revision scored for the paper with its evaluator
arguments.
The untrained model is scored the same way: pass the base model directory with
--checkpoint-dir (directories made by entrypoints/tools/prepare_untrained_base.py) and set the format with --require-think True --think-tag tool_call
(or --require-think False).
Score your own model#
Any model served through the model interface can be scored on a suite split with the generic evaluator. Apply the response format with environment overrides and assert it:
bash entrypoints/eval/eval.sh \ data.inference_file=<suite_dir>/<suite>/<split>.parquet \ model.config=agrimanager/model_interface/configs/vllm_offline/default.yaml \ model.path=<hf_repo_or_local_dir> \ +data.env_config_overrides.require_think=True \ +data.env_config_overrides.think_tag=tool_call \ +data.assert_env_config.require_think=True \ output.dir=<out_dir>
Hosted models work through --config-name openrouter and an OPENROUTER_API_KEY; use a
fixed model slug for comparable numbers, and report results on whole splits.
Scripted references#
OUT_ROOT=<dir> bash experiments/rebuttal_confirmatory_eval/run_e4_anchors.sh scores
no-action, the paper's scripted agronomic reference and the current one on every split, on a CPU (DSSAT splits
need the DSSAT runtime). The code has changed since the submitted figures, so the
paper's reference differs by simulator. The paper-era policies are kept in
experiments/rebuttal_confirmatory_eval/paper_references/, and the script copies them with their
sha256 next to the results.
| Simulator | Reference used in the paper |
|---|---|
| WOFOST | the earlier per-crop policy, paper_references/extension_guideline_rollout_paper.py |
| Cycles crop growth | the first implementation, paper_references/random_rollout_paper_sim.py, run by paper_sim_anchor_rollout.py |
| DSSAT | a code reconstruction of the fixed schedule, in paper_sim_anchor_rollout.py |
| Cycles crop planning | the current module (ag_heuristic) |
One scripted policy on one split:
bash entrypoints/eval/baseline_eval.sh --config-name no_action \ data.inference_file=<split.parquet> output.dir=<out_dir> wandb.enabled=false
NN baselines#
NN baselines run on the _nn suites, on a CPU. The paper's scripts:
experiments/t1_1_weather_regime_shift/run_eval_t11_{potato,chickpea}_nn_e4.shexperiments/t1_2_cross_crop_trait_shift/run_eval_t12_nn_e4.shexperiments/t1_3_cycles_gym_price_regime/run_eval_t13_nn_e4.shexperiments/t2_2_action_menu_shift/run_eval_t22_nn_{mask,target}_e4.sh
One NN checkpoint on one split:
bash entrypoints/eval/nn_eval.sh --config-name nn data.inference_file=<split> agent.path=<agent.zip> output.dir=<out>.
Across hardware#
Greedy decoding with vLLM is close to deterministic: where two tokens are nearly tied, a different GPU model,
driver or even a rerun on the same GPU can flip the choice, and that episode then diverges. Re-scoring the
paper's terminal checkpoints on the WOFOST and DSSAT targets in a later pass left 68 of 159 trained no-think
seed means bit-identical; trained seed means moved by at most 0.16 (no-think) and 0.29 (think), at most 1.7
scenario-level standard errors, and the untrained model's by up to 0.62 (paper, Appendix D). A DSSAT example from a
fresh install on one RTX 4090 (2026-10-25): T3.1 DSSAT-trained no-think r2 (step-114) on the T3.1
DSSAT split scored 8.00614849805832, equal to the table to all digits, with all 128 episodes identical to the
paper's bundle down to the raw model responses; a repeat on the same GPU diverged in one episode and scored
7.9922. Expect differences of that size when you re-score on other hardware.
Price-regime (T1.3): score the full split. Its Cycles environment draws each episode's weather from
NumPy's global random state at reset, after all environments of a split are built. The weather an episode gets
therefore depends on the whole batch. Scoring the complete 384-scenario split reproduces the paper's numbers
bit for bit; a subset (--max-samples) or a single replayed scenario gets other weather draws.