AgriManager / Docs
Website GitHub

Reproducing the paper

From checkpoint revisions to the CSV rows behind every table and figure.

Every number in the paper traces to a row of a CSV table in results/tables/, and every table traces to scenario-level records produced by scoring a named checkpoint revision on the frozen held-out test suite.

The pipeline#

pipeline
checkpoint revisions (step-N)            scripted and NN policies
          │                                        │
  e5_evaluate_checkpoint.sh            run_e4_anchors.sh, run_eval_*nn*_e4.sh
          └──────────── result bundles ────────────┘
                              │   per-scenario outputs + eval_manifest.json
      analysis/test_suite/extract_e4.py   →  one row per scenario
                              │
      analysis/test_suite/e4_tables.py    →  results/tables/*.csv
                              │
      analysis/test_suite/make_figures.py →  results/figures/
StepScriptWhat it does
Scoreentrypoints/eval/e5_evaluate_checkpoint.sh; run_e4_anchors.sh; run_eval_*nn*_e4.shone result bundle per checkpoint revision and setting (Evaluation)
Extractanalysis/test_suite/extract_e4.py --data-root <result roots> --out <dir>reads every bundle, takes the metric each split declares as primary, writes one row per scenario, and checks each split's mean against the evaluator's own summary
Tabulateanalysis/test_suite/e4_tables.py --episodes <dir>/e4_episodes.parquetselects each seed's terminal checkpoint, computes cell statistics, paired and between-method comparisons, reference gaps and normalized scores
Drawanalysis/test_suite/make_figures.pyredraws the three tier figures from the CSVs into results/figures/; the paper's final-layout figure scripts are kept with the paper sources

results/scoring_manifest.tsv lists every checkpoint revision scored for the paper with its evaluator arguments; feed it to entrypoints/tools/e4_score_worker.sh --queue to re-score. NN results are read from experiments/*/results/nn_eval_e4/, or from the directory given with --nn-root.

Per-run records

The final tables in results/tables/ combine the September 2026 scoring with the October 2026 re-scoring of the Tier 3 Cycles cells on the rebuilt suites. Part of the September per-run bundles and the scenario-level episode file were lost with a storage allocation; the aggregate tables survived. The bundles are being regenerated by re-scoring the revisions in results/scoring_manifest.tsv. Until that finishes, some cells reproduce only from the aggregate CSVs.

Paper items and their CSVs#

Paper itemCSV
Tier 1, 2, 3 figures; per-seed tablese4_by_seed.csv
Raw returns with mean and SD over seedse4_cells.csv
Reference returns, gaps, which targets normalizee4_anchors.csv
Normalized scorese4_normalized.csv
Think versus no-thinke4_paired.csv
LLM-RL versus NN baselinese4_vs_neural.csv
Unified policy versus cross-simulator specialistse4_specialist.csv
Invalid-output ratese4_by_seed.csv (invalid_rate)
Training curves (exploratory)experiment-tracker exports; outside the confirmatory pipeline

Method labels in the CSVs predate the final naming: Ag heuristic (paper) is the scripted agronomic reference used in the paper and Ag heuristic (current) the current code version.

FileOne row perKey columns
e4_by_seed.csvsetting × cell × method × arm × seed × targetobjective, metric, n_scenarios, invalid_rate
e4_cells.csvsetting × cell × method × arm × targetmean, sd, n_seeds
e4_anchors.csvtargetno_action, heuristic_paper, heuristic_current, gap_paper, think_gain, gap_usable
e4_normalized.csvarm × targetstatus, norm_mean, norm_sd, raw_mean
e4_paired.csvcomparison × targetmean_diff, sd_over_seeds, seeds_positive
e4_vs_neural.csv, e4_specialist.csvcomparison × targetmean_diff, propagated_sd, diff_in_prop_sd, all_a_seeds_above_all_b

Statistics#