GRPO Trainer

Group Relative Policy Optimization is the online RL algorithm behind reasoning models like DeepSeek-R1. For each prompt it samples a group of completions, scores them with a reward function, and pushes the policy toward the above-average completions — no value network required.

Reward functions

Unlike DPO, GRPO needs a reward function, not preference pairs. A reward function takes a batch of completions and returns a list of floats:

# Reward = 1.0 when the completion is exactly 20 characters long
def reward_len(completions, **kwargs):
    return [-abs(20 - len(c)) for c in completions]

Example

from datasets import load_dataset
from trl import GRPOConfig, GRPOTrainer

dataset = load_dataset("trl-lib/tldr", split="train")

def reward_len(completions, **kwargs):
    return [-abs(50 - len(c)) for c in completions]

trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    reward_funcs=reward_len,
    args=GRPOConfig(
        output_dir="Qwen2.5-0.5B-GRPO",
        num_generations=8,       # group size G
        max_completion_length=256,
    ),
    train_dataset=dataset,
)
trainer.train()

Verifiable rewards

For math and code, pass a rule-based verifier (e.g. checking the parsed answer against ground truth) as the reward function. You can also pass multiple reward functions and TRL will sum them — useful for combining a correctness reward with a format reward.

GRPO is compute-heavy because it samples num_generations completions per prompt. Rent a multi-GPU box from the Inferix marketplace and launch with accelerate + vLLM for fast generation.