RL training
How training works here, the options you control, and how the paper's runs were set up.
Training is multi-turn GRPO on VeRL,
launched with entrypoints/train/train.sh (default config agri_grpo). The policy
rolls out a season turn by turn and the trajectory return drives the update — see
Framework for the loop and Quick start for the
commands. The default config already enables GRPO, the reward-variance filter and per-turn training; the
response format (think or no-think) is set per dataset.
One policy across simulators#
The policy is prompt-conditioned: it reads the schema — observations, action menu, objective — from the prompt of each episode. WOFOST, Cycles, and DSSAT all reach it as text, even though their dynamics and interfaces differ, so one policy can train across all of them at once.
Each training instance is a triple (Σ, P, R) — the prompt-declared schema, the simulator
dynamics, and the matching reward — sampled from a mixed distribution; training maximizes the expected
return over that mixture:
J(θ) = E[ R(τ) ] with (Σ, P, R) ~ D_train , τ ~ π_θ( · | Σ )
The dynamics P never enter the policy — only the prompt schema Σ does — so mixing
simulators is just mixing rows in the dataset. The Tier 3 unified policy trains this way across the
three simulators. In the paper it learns DSSAT and approaches its DSSAT specialist, while the per-simulator
specialists stay ahead on the WOFOST and Cycles profit targets (Benchmark).
Training levers#
Two behaviors are specific to AgriManager. Everything else — the advantage estimator (GRPO by default, stepwise GAE available), model backbone, epochs, and GPUs — is plain training config; see Configuration.
- Reward-variance filter (on by default) — keeps only the
prompt groups with high in-group reward variance and skips uninformative batches (RAGEN-style), so updates
focus on rollouts that carry signal. Toggle with
trainer.rollout_filter.enable. When every group is filtered out for 5 consecutive steps (max_consecutive_all_filtered_steps), the run stops early; its last checkpoint is its final policy. - Think / no-think — in think mode the policy writes a short reasoning block in a
<tool_call>tag before its<answer>; in no-think mode it writes the answer alone. Set per dataset withenv.require_think. On the Qwen3-4B backbone, no-think training often stalled near no-action where the reward penalizes input use, and several no-think runs stopped early through the filter above. A single-seed Llama-3.2-3B pair reverses the think/no-think ordering, so treat this as a property of the backbone tested.
Paper runs#
Every paper run is a fixed script under experiments/<setting>/; running the script is the
experiment. Shared settings, set explicitly in the scripts:
| Item | Value |
|---|---|
| Backbone | Qwen/Qwen3-4B-Instruct-2507 |
| Algorithm | GRPO, no KL loss, actor learning rate 1e-6, 4 rollouts per prompt at temperature 1 |
| Length | 200 training steps; 400 for reward-formulation, 800 for the unified policy |
| Validation | before training and every 10 steps (monitoring only) |
| Checkpoint scored | the terminal one of each run (the stop step for a run that stopped early) |
| Seeds | three per arm; one seed sets data.seed and actor_rollout_ref.actor.data_loader_seed; the think and no-think arms of a replicate share it |
| Setting | Scripts (experiments/…) | Seeds |
|---|---|---|
| weather-regime | t1_1_weather_regime_shift/run_t11_{chickpea,potato}_{think,no_think}_r{2,3,4}_train.sh | 51, 52, 53 |
| crop-variety | t1_2_cross_crop_trait_shift/run_t12_{4,8,16}id_{think,no_think}_r{2,3,4}_train.sh | 81, 82, 83 |
| price-regime | t1_3_cycles_gym_price_regime/run_t13_price_{think,no_think}_r{1,2,3}_train.sh | 110, 111, 112 |
| observation-schema | t2_1_observation_schema_shift/run_t21_observation_{think,no_think}_r{2,3,4}_train.sh | 61, 62, 63 |
| action-menu | t2_2_action_menu_shift/run_t22_family_{a,b}_{think,no_think}_r{2,3,4}_train.sh | 71, 72, 73 |
| reward-formulation | t2_3_reward_formulation_shift/run_t23_{objectives,yield_only}_{think,no_think}_r{2,3,4}_train.sh | 91, 92, 93 |
| cross-simulator | t3_1_cross_simulator_maize_transfer/run_t31_{wofost,dssat,cycles}_{think,no_think}_r*_train.sh | 71, 72, 73 (Cycles think: 71, 72, 74) |
| unified policy | t3_2_unified_prompt_conditioned_policy/run_t32_unified_qwen3_{think,no_think}_r{2,3,4}_train.sh | 43, 44, 45 |
The Cycles-source think run with seed 73 aborted on a vLLM FlashAttention-3 kernel fault before its first
checkpoint; its replacement (r5, seed 74) ran with FlashAttention 2, recorded in its script.
Neural-network baselines#
entrypoints/train/nn_train.sh --config-name nn trains a Stable-Baselines3 policy on the numeric
observation, on a CPU. It is a fixed-interface baseline: its input layout and action head are fixed at
training time.
| Key | Values |
|---|---|
agent.type | PPO (default), MaskablePPO, A2C, DQN |
agent.n_epochs | 1: one gradient pass per batch, the paper's main comparison (n1, same interaction budget as the language-model runs); 8: appendix only (n8) |
agent.seed | 1, 2, 3 |