AgriManager / Docs
Website GitHub

Benchmark

A three-tier generalization ladder that asks how far a policy transfers.

The AgriManager benchmark asks whether a prompt-conditioned LLM-RL policy generalizes across the heterogeneity of agricultural management. It decomposes that question into a three-tier ladder of eight settings, with a shared evaluation protocol, where each tier widens the gap between training and test. Every setting is scored on a frozen held-out test suite with three training seeds per arm.

Each tier changes more of the task interface between training and test.

Tier 1

Within Schema

Weather, crop traits, or prices vary under the same task schema.

(O, A, R) unchanged · context changes (weather, crop, prices)

Fixed-interface NNLLM
Tier 2

Single-Axis Schema

One of measurements, management options, or goals changes.

One of O, A, R changes (action menus also bring their own environment wrappers)

Fixed-interface NNLLM
Tier 3

Compositional Schema

Across farms, measurements, management options, and goals differ.

Several of O, A, R change; dynamics P may change too

Fixed-interface NNLLM

A fixed-interface neural network cannot run once a shift changes its input layout or action encoding. A prompt-conditioned LLM re-reads the schema each episode.

The three-tier ladder#

The eight settings#

Scenarios gives training configurations × training rows per configuration / held-out test rows.

SettingVariesTrain → TestObjectiveScenarios
Tier 1 — Within Schema
weather-regimeweathernormal weather → drought, wet, hot, coldprofit2×1,600 / 1,280
crop-varietycrop variety4 / 8 / 16 seen crops → 4 held-out cropsprofit3×1,600 / 1,024
price-regimepricesnon-dominant band → high-corn, high-soygross revenue1,600 / 384
Tier 2 — Single-Axis Schema
observation-schemaΣ_Ofull schema → subset, rename, superset, anonymizedprofit1,600 / 1,024
action-menuΣ_AA: lnpkw, lnpk → lnw, ln, lw; B: lnpk, lw → lnpkw, lnw, lnprofit2×1,600 / 512
reward-formulationΣ_Rmulti-objective (yield, profit, water) or yield only → yield, profit, water, nutrientvaries1,600+1,602 / 512
Tier 3 — Compositional Schema (DSSAT · WOFOST · Cycles)
cross-simulatorΣ × Pone source simulator → 3×3 source–target pairsprofit3×1,600 / 386
unified policyΣ_O × Σ_A × Σ_R × Simseen schema tuples → held-out combinationsvaries3,200 / 1,156

Baselines & references#

The target policy is prompt-conditioned LLM-RL, in think and no-think variants. It is compared against:

Normalized return#

Normalized return places a policy relative to the two references, both scored on the same scenarios:

normalized return
R̃(π) = ( R(π) − R_no-action ) / | R_ref − R_no-action |

   R̃ = 0  →  matches no-action
   R̃ = ±1 →  matches the scripted agronomic reference
   R̃ > 1  →  exceeds the scripted agronomic reference

The gap R_ref − R_no-action is used as a denominator only where the reference clears no-action by at least a tenth of the think arm's gain over no-action on that target. Where it does not, the benchmark reports the gain over no-action, R(π) − R_no-action. The gap varies across targets, so compare policies by R̃ only within a target. Raw returns, both reference returns and the gap are published for every headline cell (Reproducing the paper).

Known limits of the references#

What the paper finds#

Confirmatory results are the three-seed test-suite results; results only on validation or from a single seed are exploratory. In short, on the Qwen3-4B backbone:

Setup#

LLM-RL trains Qwen3-4B-Instruct with GRPO on VeRL; the neural-network PPO baseline trains Stable-Baselines3 PPO on the numerical observation. The default objective is profit_max (terminal storage-organ yield minus input costs), with per-setting reward overrides. Each arm trains three seeds and is scored at its terminal checkpoint with greedy decoding; see RL training and Evaluation.

Tracking OOD generalization during training#

To watch generalization to held-out conditions live during training, declare several named validation sets along one OOD axis. AgriManager logs a separate curve per set, so an in-distribution id set sits next to each OOD set. These curves are for monitoring and are exploratory; they never select checkpoints. Two steps:

1. At dataset build — set the axis, and mark each held-out split as a validation set with an axis label:

yaml · dataset config
validation_axis: weather_regime          # the OOD axis
sampling:
  splits:
    train:                               # in-distribution training
      role: train
      labels: { weather_regime: id }
    val_id:                              # in-distribution validation
      role: validation
      validation_set: id
      labels: { weather_regime: id }
    val_drought:                         # one held-out regime
      role: validation
      validation_set: drought
      labels: { weather_regime: drought }
    # … val_wet, val_hot, val_cold likewise

2. At train — point data.val_sets at the built parquets and name the axis:

bash · train
bash entrypoints/train/train.sh --config-name agri_grpo \
  data.train_files=<dir>/train.parquet \
  data.val_files=null \
  data.validation_axis=weather_regime \
  +data.val_sets.id=<dir>/val_id.parquet \
  +data.val_sets.drought=<dir>/val_drought.parquet \
  # … +data.val_sets.{wet,hot,cold}=…

Training then logs one generalization curve per set under val-*-weather_regime/<set>/… (with a matching train-* mirror), so you read the in-distribution id set against each held-out regime as training runs. The same two keys work on any axis — action_menu, reward_formulation, or simulator.