GRPO Training Runs — Multi-Target (5-Model Pool)

Part of the data release for "Training Alignment Auditors via Reinforcement Learning" (ICLR 2026).

Four GRPO runs using a 5-model target pool (Gemini 3 Flash, Grok 4.1 Fast, Seed 2.0 Lite, Llama 3.3 70B, GPT-4.1-mini) with target-identity anonymization. All targets present as "PRISM-4 by Nexus Research" to prevent the auditor from identifying target family.

• mt3_2f_grpo_mt_v3_2fp/ — PV2 intragroup pairwise, 2/8 FP
• mt3_4f_grpo_mt_v3_4fp/ — PV2 intragroup pairwise, 4/8 FP
• mt3p1f_grpo_mt_v3_pw_1fp/ — Iterative pairwise vs baseline, 1/8 FP
• mt3p4f_grpo_mt_v3_pw_4fp/ — Iterative pairwise vs baseline, 4/8 FP

These appear in the multi-target ablation (cut_multi_target.png).

Layout

mt3_2f_grpo_mt_v3_2fp/     ... epoch_000 ... epoch_480 ...
mt3_4f_grpo_mt_v3_4fp/     ... epoch_000 ... epoch_473 ...
mt3p1f_grpo_mt_v3_pw_1fp/  ... epoch_000 ... epoch_438 ...
mt3p4f_grpo_mt_v3_pw_4fp/  ... epoch_000 ... epoch_439 ...

Per-run schema

Each epoch_*_results.json contains the per-epoch training summary for its run:

{
  "epoch": 320,
  "loss": 0.412,
  "mean_reward": 0.67,
  "kl_sample_train": 0.0,  // Anthropic EAP does not compute KL; always 0.
  "sample_entropy": 1.32,
  "train_entropy": 1.30,
  "snapshot_id": "ft_snap_01...",
  "trajectory_ids": ["ft_traj_...", ...],
  "rewards":    [0.7, 0.2, 0.9, ...],       // per-trajectory reward
  "advantages": [0.34, -0.48, 0.62, ...]   // GRPO-normalized advantages
  // additional fields vary by reward family
}

Snapshots

All presented snapshots (ft_snap_* IDs in the paper's compute-matched tables) have been promoted on the Anthropic Finetuning API to prevent the default 7-day expiry, and can be invoked via claude-haiku-4-5-20251001-ft_snap_<ID> on /v1/messages with the anthropic-beta: finetuning-2025-09-03 header.

Citation

@inproceedings{training-alignment-auditors-rl,
  title={Training Alignment Auditors via Reinforcement Learning},
  author={{Anonymous}},
  booktitle={ICLR},
  year={2026}
}

License

Research use only. No API keys or PII are included. Transcripts may contain LLM-fabricated fake credentials used as part of audit scenarios — these are not real secrets, they are scenario content probing quirked-target behavior around credential-adjacent inputs.