RoadmapBench A benchmark for evaluating AI coding agents on multi target, long horizon software development tasks derived from open source project version upgrades. Overview RoadmapBench contains 115 tasks spanning 17 open source repositories across 5 programming languages (Python, TypeScript, Go, Rust, C++). Each task requires an agent to implement multiple interdependent features that correspond to a real version upgrade of the target project. Task Structure Each task directory contains: Usage To run a single task: Languages & Repositories Language Tasks Python 41 TypeScript 22 C++ 20 Go 17 Rust 15 Citation Paper under review.
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy