BenchmarkThree tiers · eight settings
A Three-Tier Generalization Benchmark
AgriManager systematically evaluates how reinforcement learning policies generalize across changing environments and task interfaces. Eight settings, organized into three tiers, give a structured testbed for understanding where policy generalization succeeds and where it breaks.
Within Schema
Can a policy generalize to unseen environments when the task interface remains fixed?
Observation, action and reward schemas remain fixed. Context changes.
| Setting | Train | Test | Simulator | Rows (test) |
|---|---|---|---|---|
| Weather regime | Normal weather | Drought, wet, hot, cold | WOFOST | 1,280 |
| Crop variety | 4 / 8 / 16 seen crops | 4 held-out crops | WOFOST | 1,024 |
| Price regime | Typical prices | High corn or high soybean price | Cycles | 384 |
Single-Axis Schema
Can a policy adapt when one part of the task interface changes?
One schema axis is targeted at a time. The native action menus also involve wrapper differences.
| Setting | Train | Test | Simulator | Rows (test) |
|---|---|---|---|---|
| Observation schema | Full schema | Subset, renamed, superset, anonymized fields | WOFOST | 1,024 |
| Action menu | A: lnpkw, lnpk · B: lnpk, lw | A: lnw, ln, lw · B: lnpkw, lnw, ln (+ each family's seen menu) | WOFOST | 512 |
| Reward formulation | Multi-objective (yield, profit, water) or yield only | Yield, profit, water, nutrient | WOFOST | 512 |
Compositional Schema
Can learned policies transfer across simulators or recombine previously seen task components?
Cross-simulator transfer changes dynamics alongside the schema. Unified-policy tests withhold complete tuples while retaining component coverage.
| Setting | Train | Test | Simulator | Rows (test) |
|---|---|---|---|---|
| Cross-simulator transfer | One source simulator | 3 × 3 source–target pairs | all three | 386 |
| Unified policy | Seen schema tuples | Four held-out schema combinations | all three | 1,156 |
Settings and row counts as in the paper's Table 1; the eight test sets sum to the 6,278-row test suite.
Examples
Real scored test episodes from AgriManager policies: what the policy observed, what it wrote, and what it did.
Full episodes in DSSAT, WOFOST and Cycles
Four real scored test episodes, decision by decision. Scrub the season, press play, or click the chart. The same backbone before and after RL on a Cycles crop-planning scenario is on the Results page →
What a schema shift looks like in the prompt
Excerpts from real AgriManager prompts for one WOFOST maize season. An illustration of the interface, not a result.
Evaluation protocol
Frozen held-out test suite
All headline scores come from a prespecified suite of 6,278 evaluation rows, built after submission and disjoint from each experiment's training and validation data. Rows are not independent physical scenarios.
Three seeds, terminal checkpoints
Each LLM-RL arm and neural baseline trains three seeds. Every run is scored at its terminal checkpoint with no selection on validation; we report mean ± SD over seeds.
Normalized return
R̃(π) = (R(π) − Rno-action) / |Rref − Rno-action|
No-action sits at 0 and the scripted agronomic reference at 1. The gap varies across targets, so policies are compared by R̃ only within a target.
Baselines
The untrained model (the same backbone before RL), neural-network PPO with the interaction budget of LLM-RL on the Tier 1 settings, the scripted agronomic reference, and no-action.
Confirmatory and exploratory results
Three-seed test-suite results are confirmatory. Results only on validation or from a single seed (training curves, backbone comparisons) and the per-crop breakdown are exploratory and labeled as such.
Further details are in the documentation →