Supported models & algorithms
What you can plug in, what the paper ran, and how to switch.
AgriManager trains language-model policies on VeRL (docs), pinned as a submodule at v0.7.0. Rollouts run on vLLM (docs). AgriManager adds the environments, the agent loop and the reward function, and leaves VeRL unmodified. So in principle any model and RL algorithm that VeRL and vLLM support can be plugged in:
- Models: the VeRL feature list (Hugging Face Transformers models such as Qwen, Llama, Gemma and DeepSeek), adding a model to the FSDP backend, and vLLM's supported models.
- Algorithms: the same feature list (PPO, GRPO, RLOO, REINFORCE++, DAPO, DrGRPO and others), VeRL's algorithm baselines, and the GRPO and PPO pages.
Those pages track VeRL's latest release. Check a feature against v0.7.0 before relying on it.
What we ran#
Paper results means three training seeds scored on the held-out test suite. Exploratory means one seed, reported as exploratory in the paper appendix.
| Backbone | Algorithm | Status | Launcher (experiments/…) |
|---|---|---|---|
Qwen/Qwen3-4B-Instruct-2507 | GRPO | paper results, every setting | the per-setting run_*_r{2,3,4}_train.sh scripts (RL training) |
meta-llama/Llama-3.2-3B-Instruct | GRPO | exploratory, unified policy | t3_2_…/run_t32_unified_llama32_{think,no_think}_train.sh |
Qwen/Qwen2.5-3B-Instruct | GRPO | exploratory, unified policy | t3_2_…/run_t32_backbone_sweep_train.sh |
Qwen/Qwen3-8B | GRPO | exploratory, unified policy (think arm) | |
Qwen/Qwen3-4B-Instruct-2507 | DrGRPO, PPO | exploratory, unified policy |
Here t3_2_… stands for t3_2_unified_prompt_conditioned_policy. The Qwen3-8B no-think arm
stopped at step 493 and was not published.
Neural-network baselines use Stable-Baselines3
through entrypoints/train/nn_train.sh, selected with agent.type
(agrimanager/nn_ppo/algorithms.py):
agent.type | Where in the paper |
|---|---|
PPO | paper results, Tier 1 (one gradient pass per batch; eight passes in the appendix) |
MaskablePPO | action-menu setting, appendix (sb3-contrib) |
DQN, A2C | potato weather-regime, appendix |
Switch the backbone#
Set the Hugging Face model id or a local path:
bash entrypoints/train/train.sh \ actor_rollout_ref.model.path=Qwen/Qwen2.5-3B-Instruct \ data.train_files=<train.parquet> data.val_files=<val.parquet>
Prompts use the model's own chat template. To override it, set
actor_rollout_ref.model.custom_chat_template; Gemma models get a template with a system turn
automatically (agrimanager/adapter/trainer/main_ppo.py). Larger models need more GPUs
(trainer.n_gpus_per_node) or a lower actor_rollout_ref.rollout.gpu_memory_utilization.
Switch the algorithm#
| Algorithm | How |
|---|---|
| GRPO (default) | --config-name agri_grpo (algorithm.adv_estimator=grpo) |
| PPO | --config-name agri_ppo: a critic with GAE (algorithm.adv_estimator=gae); the sweep used 8 rollouts per prompt |
| DrGRPO | on agri_grpo: algorithm.norm_adv_by_std_in_grpo=False and actor_rollout_ref.actor.loss_agg_mode=seq-mean-token-sum |
| Other VeRL estimators | algorithm.adv_estimator=<name>, as documented by VeRL; not tested with AgriManager |
The sweep launcher wraps all three:
bash experiments/t3_2_unified_prompt_conditioned_policy/run_t32_backbone_sweep_train.sh \ --algo grpo|drgrpo|ppo --think think|tool_call|no_think --model <hf-model-id>
Response format#
Each arm trains and is scored in one format. Set it in the dataset config under env:, or apply it
at launch as the launchers do:
+data.env_config_overrides.require_think=True # False for no-think +data.env_config_overrides.thinking_mode=think +data.env_config_overrides.think_tag=tool_call # or think
- The paper's Qwen3-4B runs and the Llama-3.2 pair use the
<tool_call>tag with 512 response tokens. - The Qwen2.5-3B and Qwen3-8B think arms use the
<think>tag; the Qwen2.5-3B think arm also used 1,024 response tokens (data.max_response_length). Those pairs differ in more than the format. - Score a checkpoint in the format it trained with; see Evaluation.
API models (zero-shot)#
Hosted models can be scored without training through OpenRouter. Configs live in
agrimanager/model_interface/configs/openrouter/; set OPENROUTER_API_KEY and pick a
model slug from openrouter.ai/models:
bash entrypoints/eval/eval.sh --config-name openrouter \ data.inference_file=<split.parquet> model.path=<model-slug> output.dir=<out_dir>
These evaluations are not part of the paper results. Use a fixed model slug for comparable numbers.