4-Bucket Evaluation Suite — Full Transcripts

Part of the data release for "Training Alignment Auditors via Reinforcement Learning" (ICLR 2026).

Full audit transcripts + judge outputs for all 80 auditor configurations evaluated on the paper's 4-bucket evaluation suite (Cells A/B/E/F/G). The results_no_thinking/ snapshot is the primary source for all paper figures that reference composite scores (ranking.png, frontier_comparison.png, scaling_*.png).

Cells:

• Cell A — 6 held-out quirks × 15 tailored seeds × 3-5 rollouts vs DeepSeek v3.1 MO (investigation depth)
• Cell B — same target × 181 Petri seeds × 3 rollouts (breadth; correlated with A)
• Cell E — 6 held-out quirks × 15 tailored seeds × 3-5 rollouts vs clean Sonnet 4.6 (FPR calibration)
• Cell F — 181 Petri seeds × 3-5 rollouts vs real Sonnet 4.5 (production value)
• Cell G — 20 realism seeds × 5 rollouts, NIAH pairwise vs WildChat

All judged by Opus 4.6 (no thinking) with the custom eval-suite rubric.

Layout

polished_eval/results_no_thinking/
├── haiku_4_5/
│   ├── cell_a/ {config.json, results.jsonl, judged_results.jsonl, summary.json, transcripts/*.json}
│   ├── cell_b/ {same structure}
│   ├── cell_e/ {same structure}
│   └── cell_f/ {same structure}
├── sonnet_4_5/ ...
├── sonnet_4_6/ ...
├── opus_4_5/, opus_4_6/, opus_4_7/ ...
├── gpt_5_4/, gpt_5_4_mini/, gpt_5_4_nano/ ...
├── gemini_3_flash/, gemini_3_1_flash_lite/, gemini_3_1_pro/ ...
├── a1f_ep160/, a1f_ep325/ ...
├── a2f_ep160/, a2f_ep325/, a2f_ep490/, a2f_ep650/ ...
├── a4f_ep160/, ..., a4f_ep650/
├── apr_ep160/, apr_ep325/, apr_ep490/
├── apw_ep160/, apw_ep325/
├── ap4_ep160/, ap4_ep324/, ap4_ep490/  (the headline model)
├── mt3_2f_ep160/, mt3_2f_ep325/, mt3_2f_ep480/
├── mt3_4f_ep160/, mt3_4f_ep325/
├── mt3p1f_ep160/, mt3p1f_ep325/
├── mt3p4f_ep160/, mt3p4f_ep325/
├── ig_scaled_ep160/, ig_scaled_ep325/, ig_scaled_ep490/
├── ig_petri_ep160/, ig_petri_ep325/, ig_petri_ep490/
├── ig_pv2_ep160/, ig_pv2_ep325/, ig_pv2_ep480/
├── pw_grok_fixed_ep325/, pw_grok_fixed_ep490/
├── pw_gpt4m_fixed_ep490/
├── pw_iter_grok_ep325/, pw_iter_grok_ep485/
├── binary_v1_ep160/, binary_v1_ep325/, binary_v1_ep480/
├── binary_v2_ep160/, binary_v2_ep325/
└── conc_8t_ep80/, conc_8t_ep163/, conc_8t_ep240/

Citation

@inproceedings{training-alignment-auditors-rl,
  title={Training Alignment Auditors via Reinforcement Learning},
  author={{Anonymous}},
  booktitle={ICLR},
  year={2026}
}

License

Research use only. No API keys or PII are included. Transcripts may contain LLM-fabricated fake credentials used as part of audit scenarios — these are not real secrets, they are scenario content probing quirked-target behavior.