ITBench AA Artificial Analysis' release of the public scenarios from IBM's ITBench benchmark, used for the ITBench AA leaderboard. This repo currently contains the SRE subset ( sre config). Each row is a Kubernetes incident scenario with its expected contributing factor entities. An agent under evaluation is given access to an offline snapshot of the affected cluster (alerts, events, traces, topology) and must identify the entity (Deployment, Pod, ConfigMap, etc.) responsible for the failure. The sre config contains only the 40 scenarios marked source split: public in our internal dataset (out of 59 total). The remaining private/held out scenarios are not included. Source The scenarios originate from itbench hub/ITBench . See the per scenario directories at scenarios/sre/project/roles/scenarios/files/scenario in that repo. Fields id aa — Artificial Analysis row identifier scenario id — upstream scenario identifier (e.g. Scenario 3 ) category — module identifier ( sre ) source split — always public in this release scenario root — filesystem root used inside the evaluation sandbox ground truth yaml — YAML describing the fault, alerts, entity groups, propagation graph, and recommended…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy