AgriManager / Docs
Website GitHub

RL training

How training works here, the options you control, and how the paper's runs were set up.

Training is multi-turn GRPO on VeRL, launched with entrypoints/train/train.sh (default config agri_grpo). The policy rolls out a season turn by turn and the trajectory return drives the update — see Framework for the loop and Quick start for the commands. The default config already enables GRPO, the reward-variance filter and per-turn training; the response format (think or no-think) is set per dataset.

One policy across simulators#

The policy is prompt-conditioned: it reads the schema — observations, action menu, objective — from the prompt of each episode. WOFOST, Cycles, and DSSAT all reach it as text, even though their dynamics and interfaces differ, so one policy can train across all of them at once.

Each training instance is a triple (Σ, P, R) — the prompt-declared schema, the simulator dynamics, and the matching reward — sampled from a mixed distribution; training maximizes the expected return over that mixture:

joint objective
J(θ) = E[ R(τ) ]     with   (Σ, P, R) ~ D_train ,   τ ~ π_θ( · | Σ )

The dynamics P never enter the policy — only the prompt schema Σ does — so mixing simulators is just mixing rows in the dataset. The Tier 3 unified policy trains this way across the three simulators. In the paper it learns DSSAT and approaches its DSSAT specialist, while the per-simulator specialists stay ahead on the WOFOST and Cycles profit targets (Benchmark).

Training levers#

Two behaviors are specific to AgriManager. Everything else — the advantage estimator (GRPO by default, stepwise GAE available), model backbone, epochs, and GPUs — is plain training config; see Configuration.

Paper runs#

Every paper run is a fixed script under experiments/<setting>/; running the script is the experiment. Shared settings, set explicitly in the scripts:

ItemValue
BackboneQwen/Qwen3-4B-Instruct-2507
AlgorithmGRPO, no KL loss, actor learning rate 1e-6, 4 rollouts per prompt at temperature 1
Length200 training steps; 400 for reward-formulation, 800 for the unified policy
Validationbefore training and every 10 steps (monitoring only)
Checkpoint scoredthe terminal one of each run (the stop step for a run that stopped early)
Seedsthree per arm; one seed sets data.seed and actor_rollout_ref.actor.data_loader_seed; the think and no-think arms of a replicate share it
SettingScripts (experiments/…)Seeds
weather-regimet1_1_weather_regime_shift/run_t11_{chickpea,potato}_{think,no_think}_r{2,3,4}_train.sh51, 52, 53
crop-varietyt1_2_cross_crop_trait_shift/run_t12_{4,8,16}id_{think,no_think}_r{2,3,4}_train.sh81, 82, 83
price-regimet1_3_cycles_gym_price_regime/run_t13_price_{think,no_think}_r{1,2,3}_train.sh110, 111, 112
observation-schemat2_1_observation_schema_shift/run_t21_observation_{think,no_think}_r{2,3,4}_train.sh61, 62, 63
action-menut2_2_action_menu_shift/run_t22_family_{a,b}_{think,no_think}_r{2,3,4}_train.sh71, 72, 73
reward-formulationt2_3_reward_formulation_shift/run_t23_{objectives,yield_only}_{think,no_think}_r{2,3,4}_train.sh91, 92, 93
cross-simulatort3_1_cross_simulator_maize_transfer/run_t31_{wofost,dssat,cycles}_{think,no_think}_r*_train.sh71, 72, 73 (Cycles think: 71, 72, 74)
unified policyt3_2_unified_prompt_conditioned_policy/run_t32_unified_qwen3_{think,no_think}_r{2,3,4}_train.sh43, 44, 45

The Cycles-source think run with seed 73 aborted on a vLLM FlashAttention-3 kernel fault before its first checkpoint; its replacement (r5, seed 74) ran with FlashAttention 2, recorded in its script.

Neural-network baselines#

entrypoints/train/nn_train.sh --config-name nn trains a Stable-Baselines3 policy on the numeric observation, on a CPU. It is a fixed-interface baseline: its input layout and action head are fixed at training time.

KeyValues
agent.typePPO (default), MaskablePPO, A2C, DQN
agent.n_epochs1: one gradient pass per batch, the paper's main comparison (n1, same interaction budget as the language-model runs); 8: appendix only (n8)
agent.seed1, 2, 3