Reproducing the paper
From checkpoint revisions to the CSV rows behind every table and figure.
Every number in the paper traces to a row of a CSV table in results/tables/, and every table traces to scenario-level
records produced by scoring a named checkpoint revision on the frozen held-out test
suite.
The pipeline#
checkpoint revisions (step-N) scripted and NN policies
│ │
e5_evaluate_checkpoint.sh run_e4_anchors.sh, run_eval_*nn*_e4.sh
└──────────── result bundles ────────────┘
│ per-scenario outputs + eval_manifest.json
analysis/test_suite/extract_e4.py → one row per scenario
│
analysis/test_suite/e4_tables.py → results/tables/*.csv
│
analysis/test_suite/make_figures.py → results/figures/
| Step | Script | What it does |
|---|---|---|
| Score | entrypoints/eval/e5_evaluate_checkpoint.sh; run_e4_anchors.sh; run_eval_*nn*_e4.sh | one result bundle per checkpoint revision and setting (Evaluation) |
| Extract | analysis/test_suite/extract_e4.py --data-root <result roots> --out <dir> | reads every bundle, takes the metric each split declares as primary, writes one row per scenario, and checks each split's mean against the evaluator's own summary |
| Tabulate | analysis/test_suite/e4_tables.py --episodes <dir>/e4_episodes.parquet | selects each seed's terminal checkpoint, computes cell statistics, paired and between-method comparisons, reference gaps and normalized scores |
| Draw | analysis/test_suite/make_figures.py | redraws the three tier figures from the CSVs into results/figures/; the paper's final-layout figure scripts are kept with the paper sources |
results/scoring_manifest.tsv lists every checkpoint revision scored for the paper with its evaluator
arguments; feed it to entrypoints/tools/e4_score_worker.sh --queue to re-score. NN results are read
from experiments/*/results/nn_eval_e4/, or from the directory given with --nn-root.
The final tables in results/tables/ combine the September 2026 scoring with the October 2026
re-scoring of the Tier 3 Cycles cells on the rebuilt suites. Part of the September per-run bundles and the
scenario-level episode file were lost with a storage allocation; the aggregate tables survived. The bundles are
being regenerated by re-scoring the revisions in results/scoring_manifest.tsv. Until that finishes,
some cells reproduce only from the aggregate CSVs.
Paper items and their CSVs#
| Paper item | CSV |
|---|---|
| Tier 1, 2, 3 figures; per-seed tables | e4_by_seed.csv |
| Raw returns with mean and SD over seeds | e4_cells.csv |
| Reference returns, gaps, which targets normalize | e4_anchors.csv |
| Normalized scores | e4_normalized.csv |
| Think versus no-think | e4_paired.csv |
| LLM-RL versus NN baselines | e4_vs_neural.csv |
| Unified policy versus cross-simulator specialists | e4_specialist.csv |
| Invalid-output rates | e4_by_seed.csv (invalid_rate) |
| Training curves (exploratory) | experiment-tracker exports; outside the confirmatory pipeline |
Method labels in the CSVs predate the final naming: Ag heuristic (paper) is the scripted
agronomic reference used in the paper and Ag heuristic (current) the current code version.
| File | One row per | Key columns |
|---|---|---|
e4_by_seed.csv | setting × cell × method × arm × seed × target | objective, metric, n_scenarios, invalid_rate |
e4_cells.csv | setting × cell × method × arm × target | mean, sd, n_seeds |
e4_anchors.csv | target | no_action, heuristic_paper, heuristic_current, gap_paper, think_gain, gap_usable |
e4_normalized.csv | arm × target | status, norm_mean, norm_sd, raw_mean |
e4_paired.csv | comparison × target | mean_diff, sd_over_seeds, seeds_positive |
e4_vs_neural.csv, e4_specialist.csv | comparison × target | mean_diff, propagated_sd, diff_in_prop_sd, all_a_seeds_above_all_b |
Statistics#
- Cells. Mean and SD over the three training seeds of an arm; each seed's value is its mean over the scenarios of the split.
- Think versus no-think. Paired by seed (the arms share the training seed) and by scenario.
- Between methods (LLM-RL versus NN, unified versus specialist). Seeds are treated as independent; the
difference is reported in units of the propagated SD,
sqrt(sd_a² + sd_b²). - Confirmatory and exploratory. Three-seed test-suite results are confirmatory. Results only on validation or from one seed are exploratory and labelled as such, for example training curves and the backbone comparisons.