DPO Trainer

Direct Preference Optimization aligns a model to human preferences without a separate reward model or online sampling. Given pairs of chosen and rejected responses, DPO directly increases the relative log-probability of the preferred answer.

Dataset format

DPO needs a preference dataset with three columns: prompt, chosen, rejected.

{
  "prompt":   "Explain gravity to a five year old.",
  "chosen":   "Gravity is what pulls things down to the ground...",
  "rejected": "Gravity is the fundamental interaction described by general relativity...",
}

Example

from datasets import load_dataset
from trl import DPOConfig, DPOTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

trainer = DPOTrainer(
    model=model,
    args=DPOConfig(output_dir="Qwen2.5-0.5B-DPO", beta=0.1),
    train_dataset=dataset,
    processing_class=tokenizer,
)
trainer.train()

The beta hyperparameter

beta controls how far the policy may drift from the reference model. Lower beta (e.g. 0.1) allows larger updates; higher beta keeps the model closer to its SFT starting point. Start at 0.1 and tune from there.

Loss variants

Set loss_type in DPOConfig to switch objectives: "sigmoid" (vanilla DPO), "ipo", "hinge", or "kto_pair".