RoadmapBench

A benchmark for evaluating AI coding agents on multi-target, long-horizon software development tasks derived from open-source project version upgrades.

Overview

RoadmapBench contains 115 tasks spanning 17 open-source repositories across 5 programming languages (Python, TypeScript, Go, Rust, C++). Each task requires an agent to implement multiple interdependent features that correspond to a real version upgrade of the target project.

Task Structure

Each task directory contains:

{task-slug}/
├── environment/
│   ├── Dockerfile        # Reproducible build environment
│   └── repo/             # Source code snapshot at V_OLD tag
├── solution/
│   ├── changes.patch     # Oracle patch (V_OLD → V_NEW diff)
│   └── solve.sh          # Script to apply the oracle solution
├── tests/
│   ├── test.sh           # Test runner (outputs reward.txt)
│   └── test_*.{go,py,rs,...}  # Per-target test files
├── instruction.md        # Multi-target development roadmap
└── task.toml             # Task metadata and resource limits

Usage

To run a single task:

cd <task-slug>
docker build -t <tag> ./environment/
docker run --rm -v ./tests:/tests <tag> bash /tests/test.sh

Languages & Repositories

Citation

Paper under review.