Dataset Summary SWE Bench Pro is a challenging, enterprise level dataset for testing agent ability on long horizon software engineering tasks. Paper: https://static.scale.com/uploads/654197dc94d34f66c0f5184e/SWEAP Eval Scale%20(9).pdf See the related evaluation Github: https://github.com/scaleapi/SWE bench Pro os Dataset Structure We follow SWE Bench Verified (https://huggingface.co/datasets/SWE bench/SWE bench Verified) in terms of dataset structure, with several extra fields. Data Fields repo (string): Repository identifier one of 11 repository classes instance id (string): Unique identifier for each instance (65 120 characters) base commit (string): Git commit hash of the base version (40 characters) patch (string): The golden code patch/diff (1.44k 180k characters) test patch (string): Test cases related to the patch (325 322k characters) problem statement (string): Description of the issue being addressed (419 8.04k characters) requirements (string): Project requirements or dependencies (124 6.7k characters, may be null) interface (string): API or interface specifications (1 12.2k characters, may be null) repo language (string): Programming language of the repository one of 4…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy