AgriManager / Docs
Website GitHub

Supported models & algorithms

What you can plug in, what the paper ran, and how to switch.

AgriManager trains language-model policies on VeRL (docs), pinned as a submodule at v0.7.0. Rollouts run on vLLM (docs). AgriManager adds the environments, the agent loop and the reward function, and leaves VeRL unmodified. So in principle any model and RL algorithm that VeRL and vLLM support can be plugged in:

Those pages track VeRL's latest release. Check a feature against v0.7.0 before relying on it.

What we ran#

Paper results means three training seeds scored on the held-out test suite. Exploratory means one seed, reported as exploratory in the paper appendix.

BackboneAlgorithmStatusLauncher (experiments/…)
Qwen/Qwen3-4B-Instruct-2507GRPOpaper results, every settingthe per-setting run_*_r{2,3,4}_train.sh scripts (RL training)
meta-llama/Llama-3.2-3B-InstructGRPOexploratory, unified policyt3_2_…/run_t32_unified_llama32_{think,no_think}_train.sh
Qwen/Qwen2.5-3B-InstructGRPOexploratory, unified policyt3_2_…/run_t32_backbone_sweep_train.sh
Qwen/Qwen3-8BGRPOexploratory, unified policy (think arm)
Qwen/Qwen3-4B-Instruct-2507DrGRPO, PPOexploratory, unified policy

Here t3_2_… stands for t3_2_unified_prompt_conditioned_policy. The Qwen3-8B no-think arm stopped at step 493 and was not published.

Neural-network baselines use Stable-Baselines3 through entrypoints/train/nn_train.sh, selected with agent.type (agrimanager/nn_ppo/algorithms.py):

agent.typeWhere in the paper
PPOpaper results, Tier 1 (one gradient pass per batch; eight passes in the appendix)
MaskablePPOaction-menu setting, appendix (sb3-contrib)
DQN, A2Cpotato weather-regime, appendix

Switch the backbone#

Set the Hugging Face model id or a local path:

bash
bash entrypoints/train/train.sh \
  actor_rollout_ref.model.path=Qwen/Qwen2.5-3B-Instruct \
  data.train_files=<train.parquet> data.val_files=<val.parquet>

Prompts use the model's own chat template. To override it, set actor_rollout_ref.model.custom_chat_template; Gemma models get a template with a system turn automatically (agrimanager/adapter/trainer/main_ppo.py). Larger models need more GPUs (trainer.n_gpus_per_node) or a lower actor_rollout_ref.rollout.gpu_memory_utilization.

Switch the algorithm#

AlgorithmHow
GRPO (default)--config-name agri_grpo (algorithm.adv_estimator=grpo)
PPO--config-name agri_ppo: a critic with GAE (algorithm.adv_estimator=gae); the sweep used 8 rollouts per prompt
DrGRPOon agri_grpo: algorithm.norm_adv_by_std_in_grpo=False and actor_rollout_ref.actor.loss_agg_mode=seq-mean-token-sum
Other VeRL estimatorsalgorithm.adv_estimator=<name>, as documented by VeRL; not tested with AgriManager

The sweep launcher wraps all three:

bash
bash experiments/t3_2_unified_prompt_conditioned_policy/run_t32_backbone_sweep_train.sh \
  --algo grpo|drgrpo|ppo --think think|tool_call|no_think --model <hf-model-id>

Response format#

Each arm trains and is scored in one format. Set it in the dataset config under env:, or apply it at launch as the launchers do:

hydra overrides
+data.env_config_overrides.require_think=True      # False for no-think
+data.env_config_overrides.thinking_mode=think
+data.env_config_overrides.think_tag=tool_call     # or think

API models (zero-shot)#

Hosted models can be scored without training through OpenRouter. Configs live in agrimanager/model_interface/configs/openrouter/; set OPENROUTER_API_KEY and pick a model slug from openrouter.ai/models:

bash
bash entrypoints/eval/eval.sh --config-name openrouter \
  data.inference_file=<split.parquet> model.path=<model-slug> output.dir=<out_dir>

These evaluations are not part of the paper results. Use a fixed model slug for comparable numbers.