TRL — Transformer Reinforcement Learning

TRL is a full-stack library for post-training language models with techniques such as Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), Group Relative Policy Optimization (GRPO), reward modeling, and Proximal Policy Optimization (PPO).

It is built on top of transformers and integrates with accelerate, peft, and datasets, so you can scale from a single GPU to a multi-node cluster without rewriting your training loop.

Why TRL

• One trainer per method. SFTTrainer, DPOTrainer, GRPOTrainer, RewardTrainer, and PPOTrainer share the same transformers.Trainer API, so you already know how to use them.
• PEFT-native. Pass a LoraConfig and train adapters instead of full weights.
• Scales out. Any TRL script runs under accelerate launch with DeepSpeed ZeRO-3 or FSDP.
• Runs on Inferix GPUs. Launch a fine-tune job from AutoTrain or rent a raw GPU and run these scripts directly.

The post-training recipe

Most modern instruction-tuned models follow a three-stage recipe, each with a matching TRL trainer:

1. SFT — teach the base model to follow instructions on a labelled dataset (SFTTrainer).
2. Reward modeling (optional) — train a model to score responses from preference pairs (RewardTrainer).
3. Preference optimization — align the SFT model to human preferences with DPOTrainer (offline) or GRPOTrainer/PPOTrainer (online RL).

Continue to Installation to set up your environment.