Within Schema
Weather, crop traits, or prices vary under the same task schema.
(O, A, R) unchanged · context changes (weather, crop, prices)
A three-tier generalization ladder that asks how far a policy transfers.
The AgriManager benchmark asks whether a prompt-conditioned LLM-RL policy generalizes across the heterogeneity of agricultural management. It decomposes that question into a three-tier ladder of eight settings, with a shared evaluation protocol, where each tier widens the gap between training and test. Every setting is scored on a frozen held-out test suite with three training seeds per arm.
Each tier changes more of the task interface between training and test.
Weather, crop traits, or prices vary under the same task schema.
(O, A, R) unchanged · context changes (weather, crop, prices)
One of measurements, management options, or goals changes.
One of O, A, R changes (action menus also bring their own environment wrappers)
Across farms, measurements, management options, and goals differ.
Several of O, A, R change; dynamics P may change too
A fixed-interface neural network cannot run once a shift changes its input layout or action encoding. A prompt-conditioned LLM re-reads the schema each episode.
Σ_O, action menu Σ_A, or reward formulation Σ_R changes between train and test.P across DSSAT, WOFOST, and Cycles.Scenarios gives training configurations × training rows per configuration / held-out test rows.
| Setting | Varies | Train → Test | Objective | Scenarios |
|---|---|---|---|---|
| Tier 1 — Within Schema | ||||
| weather-regime | weather | normal weather → drought, wet, hot, cold | profit | 2×1,600 / 1,280 |
| crop-variety | crop variety | 4 / 8 / 16 seen crops → 4 held-out crops | profit | 3×1,600 / 1,024 |
| price-regime | prices | non-dominant band → high-corn, high-soy | gross revenue | 1,600 / 384 |
| Tier 2 — Single-Axis Schema | ||||
| observation-schema | Σ_O | full schema → subset, rename, superset, anonymized | profit | 1,600 / 1,024 |
| action-menu | Σ_A | A: lnpkw, lnpk → lnw, ln, lw; B: lnpk, lw → lnpkw, lnw, ln | profit | 2×1,600 / 512 |
| reward-formulation | Σ_R | multi-objective (yield, profit, water) or yield only → yield, profit, water, nutrient | varies | 1,600+1,602 / 512 |
| Tier 3 — Compositional Schema (DSSAT · WOFOST · Cycles) | ||||
| cross-simulator | Σ × P | one source simulator → 3×3 source–target pairs | profit | 3×1,600 / 386 |
| unified policy | Σ_O × Σ_A × Σ_R × Sim | seen schema tuples → held-out combinations | varies | 3,200 / 1,156 |
The target policy is prompt-conditioned LLM-RL, in think and no-think variants. It is compared against:
Normalized return places a policy relative to the two references, both scored on the same scenarios:
R̃(π) = ( R(π) − R_no-action ) / | R_ref − R_no-action | R̃ = 0 → matches no-action R̃ = ±1 → matches the scripted agronomic reference R̃ > 1 → exceeds the scripted agronomic reference
The gap R_ref − R_no-action is used as a denominator only where the reference clears no-action by
at least a tenth of the think arm's gain over no-action on that target. Where it does not, the benchmark
reports the gain over no-action, R(π) − R_no-action. The gap varies across targets, so
compare policies by R̃ only within a target. Raw returns, both reference returns and the gap are
published for every headline cell (Reproducing the paper).
ag_heuristic code has no nitrogen rule on the Cycles maize task. The paper uses the first
implementation, which reproduces the submitted reference exactly; see
Evaluation for how each paper reference is run.Confirmatory results are the three-seed test-suite results; results only on validation or from a single seed are exploratory. In short, on the Qwen3-4B backbone:
LLM-RL trains Qwen3-4B-Instruct with GRPO on VeRL; the neural-network PPO baseline trains Stable-Baselines3 PPO on
the numerical observation. The default objective is profit_max (terminal storage-organ yield minus
input costs), with per-setting reward overrides. Each arm trains three seeds and is scored at its terminal
checkpoint with greedy decoding; see RL training and
Evaluation.
To watch generalization to held-out conditions live during training, declare several named
validation sets along one OOD axis. AgriManager logs a separate curve per set, so an
in-distribution id set sits next to each OOD set. These curves are for monitoring and are
exploratory; they never select checkpoints. Two steps:
1. At dataset build — set the axis, and mark each held-out split as a validation set with an axis label:
validation_axis: weather_regime # the OOD axis sampling: splits: train: # in-distribution training role: train labels: { weather_regime: id } val_id: # in-distribution validation role: validation validation_set: id labels: { weather_regime: id } val_drought: # one held-out regime role: validation validation_set: drought labels: { weather_regime: drought } # … val_wet, val_hot, val_cold likewise
2. At train — point data.val_sets at the built parquets and name the axis:
bash entrypoints/train/train.sh --config-name agri_grpo \ data.train_files=<dir>/train.parquet \ data.val_files=null \ data.validation_axis=weather_regime \ +data.val_sets.id=<dir>/val_id.parquet \ +data.val_sets.drought=<dir>/val_drought.parquet \ # … +data.val_sets.{wet,hot,cold}=…
Training then logs one generalization curve per set under
val-*-weather_regime/<set>/… (with a matching train-* mirror), so you read
the in-distribution id set against each held-out regime as training runs. The same two
keys work on any axis — action_menu, reward_formulation, or
simulator.