AgriManager / Docs
Website GitHub

Evaluation

Score a checkpoint, your own model, a scripted reference or an NN baseline on the held-out test suite.

Protocol#

Every number in the paper follows these rules. Follow them to get comparable numbers.

RuleDetail
Frozen test suitescore on the held-out test suite; no training, checkpoint selection, prompt tuning or decoding change uses its scenarios
Terminal checkpointscore the last checkpoint of each run (the stop step for a run that stopped early), with no selection on validation
Training response formata think-trained checkpoint is scored in think format and a no-think one in no-think format; the evaluator reads the format from the checkpoint and asserts it
Greedy decodingruntime.temperature=0, one pass (runtime.num_trials=1)
No turn capleave runtime.max_turns unset for any reported run
Three seedsreport mean ± SD over three training seeds per arm, and differences between arms in propagated SD
Raw and referencesreport the raw return with both reference returns and the gap; normalize only where the gap is usable (Benchmark)

Score a released checkpoint#

entrypoints/eval/e5_evaluate_checkpoint.sh scores one checkpoint revision on every suite group of a setting and writes one result bundle. It needs one CUDA GPU and install.sh --gpu.

bash
bash entrypoints/eval/e5_evaluate_checkpoint.sh \
  --checkpoint-repo AgriManager-Framework/schema-t32-qwen-think-r2 \
  --revision step-800 \
  --suite-repo AgriManager-Framework/agrimanager-test-suite \
  --results-repo <your-user>/<results-dataset> \
  --tier t32 \
  --no-upload
ArgumentMeaning
--checkpoint-repo, --revisionHugging Face model repository and branch step-N; the terminal revision is the highest N (Models and data)
--checkpoint-dirscore weights already on disk (used for the untrained model)
--suite-repotest-suite dataset repository (required); with --suite-dir it only names the suite in the bundle
--suite-dirscore suites already on disk (a directory holding the suite-group folders)
--results-reporesults dataset for the upload (required, also with --no-upload, where it is unused)
--tiert11, t11_chickpea, t12, t13, t21, t22, t23, t31, t32
--suite-groupoverride the setting's groups (repeatable), for example to score a cross-simulator specialist on the unified-policy suites
--require-think, --think-tagset the response format by hand, for checkpoints whose manifest does not record it
--max-samples Nscore the first N rows of each split, for a pipeline check
--no-uploadkeep the bundle local, in <work-dir>/bundle (--work-dir, default ./.e5_work); without it, HF_TOKEN is required

The bundle holds per-scenario outputs, the metrics of every split, any failures, the applied environment overrides, and eval_manifest.json, which ties every number to the checkpoint and suite revisions that produced it.

WOFOST suites read weather from a public weather pool, and the evaluators download the pool the first time a suite needs it: the maize pool (0.25 GB) for Tier 2 and the WOFOST parts of Tier 3, the 20-crop pool (17.6 GB) for weather-regime and crop-variety. The rebuilt Cycles suites are scored with --suite-group; see Held-out test suite. results/scoring_manifest.tsv lists every revision scored for the paper with its evaluator arguments.

The untrained model is scored the same way: pass the base model directory with --checkpoint-dir (directories made by entrypoints/tools/prepare_untrained_base.py) and set the format with --require-think True --think-tag tool_call (or --require-think False).

Score your own model#

Any model served through the model interface can be scored on a suite split with the generic evaluator. Apply the response format with environment overrides and assert it:

bash
bash entrypoints/eval/eval.sh \
  data.inference_file=<suite_dir>/<suite>/<split>.parquet \
  model.config=agrimanager/model_interface/configs/vllm_offline/default.yaml \
  model.path=<hf_repo_or_local_dir> \
  +data.env_config_overrides.require_think=True \
  +data.env_config_overrides.think_tag=tool_call \
  +data.assert_env_config.require_think=True \
  output.dir=<out_dir>

Hosted models work through --config-name openrouter and an OPENROUTER_API_KEY; use a fixed model slug for comparable numbers, and report results on whole splits.

Scripted references#

OUT_ROOT=<dir> bash experiments/rebuttal_confirmatory_eval/run_e4_anchors.sh scores no-action, the paper's scripted agronomic reference and the current one on every split, on a CPU (DSSAT splits need the DSSAT runtime). The code has changed since the submitted figures, so the paper's reference differs by simulator. The paper-era policies are kept in experiments/rebuttal_confirmatory_eval/paper_references/, and the script copies them with their sha256 next to the results.

SimulatorReference used in the paper
WOFOSTthe earlier per-crop policy, paper_references/extension_guideline_rollout_paper.py
Cycles crop growththe first implementation, paper_references/random_rollout_paper_sim.py, run by paper_sim_anchor_rollout.py
DSSATa code reconstruction of the fixed schedule, in paper_sim_anchor_rollout.py
Cycles crop planningthe current module (ag_heuristic)

One scripted policy on one split:

bash · CPU
bash entrypoints/eval/baseline_eval.sh --config-name no_action \
  data.inference_file=<split.parquet> output.dir=<out_dir> wandb.enabled=false

NN baselines#

NN baselines run on the _nn suites, on a CPU. The paper's scripts:

One NN checkpoint on one split: bash entrypoints/eval/nn_eval.sh --config-name nn data.inference_file=<split> agent.path=<agent.zip> output.dir=<out>.

Across hardware#

Greedy decoding with vLLM is close to deterministic: where two tokens are nearly tied, a different GPU model, driver or even a rerun on the same GPU can flip the choice, and that episode then diverges. Re-scoring the paper's terminal checkpoints on the WOFOST and DSSAT targets in a later pass left 68 of 159 trained no-think seed means bit-identical; trained seed means moved by at most 0.16 (no-think) and 0.29 (think), at most 1.7 scenario-level standard errors, and the untrained model's by up to 0.62 (paper, Appendix D). A DSSAT example from a fresh install on one RTX 4090 (2026-10-25): T3.1 DSSAT-trained no-think r2 (step-114) on the T3.1 DSSAT split scored 8.00614849805832, equal to the table to all digits, with all 128 episodes identical to the paper's bundle down to the raw model responses; a repeat on the same GPU diverged in one episode and scored 7.9922. Expect differences of that size when you re-score on other hardware.

Price-regime (T1.3): score the full split. Its Cycles environment draws each episode's weather from NumPy's global random state at reset, after all environments of a split are built. The weather an episode gets therefore depends on the whole batch. Scoring the complete 384-scenario split reproduces the paper's numbers bit for bit; a subset (--max-samples) or a single replayed scenario gets other weather draws.