ProgramBench Generated Tests This dataset contains the AI generated behavioral test suites used to evaluate model solutions in ProgramBench. ProgramBench is a benchmark that evaluates whether language models can rebuild programs from scratch. Given only a compiled binary and its documentation, AI agents must architect and implement a complete codebase that reproduces the original program's behavior. These test suites are used to assess whether a candidate solution is behaviorally equivalent to the original program. Website Paper GitHub Leaderboard Dataset Overview The dataset covers 200 task instances spanning open source repositories in 6 programming languages: Language Count Rust 107 Go 46 C 33 C++ 12 Haskell 1 Java 1 Task difficulties: 27 easy, 120 medium, 18 hard, 35 unrated. Dataset Structure Each task instance is a folder named . containing: tests/ — Compressed archives, each containing a pytest based test suite generated by an AI model against the original program's compiled executable. Multiple archives may exist per task (from different generation runs). ATTRIBUTION.md — Attribution information including a link to the original repository at the pinned commit and the licens…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy