dataset_v3_500_2026-05-03

500-case curated RCA evaluation set, rebuilt 2026-05-03 with the manifest-driven causal-graph reasoner (rcabench-platform PR #370 / forge-rework). Same 1464-case raw pool as v1/v2 (_pool_v1_2026-05-02, sourced from FSE/openrca2 + AegisLab detector_success last 48h), but the ground-truth causal_graph.json for every case is regenerated by the new fault-kind manifest expansion instead of the legacy rule-only propagator.

What changed vs v2

v2's causal_graph.json came from the legacy rule-only propagator. That propagator produced an average of 463 candidate paths per case (35× more than the new builder) and relied on per-edge gates to drop the spurious ones at the end. The new manifest-driven path builder enumerates layer by layer from the registered fault manifest, so the graph it ships with each case is what the fault contract itself says is propagation — not whatever post-filtered out of an exhaustive walk.

The shift in graph structure is uniform: smaller mean degree, fewer hallucinated transit nodes, and 41/500 v2 cases now have cyclic service graphs under the new reasoner (graph-F1 metric becomes applicable=False for those, but the rest of the eval still scores).

We dropped those 41 cyclic + 10 newly-degenerate (lp ≤ 1) cases from v2, re-ran greedy selection on the full 1464-case pool under the new graphs, and re-pulled the top 500 by (longest_path desc, n_edge desc, n_svc desc) with the v1/v2 diversity caps re-tuned for the new pool composition.

The middle (lp = 3) tier shrunk and both the deep tail (lp ≥ 4) and the shallow tier (lp = 2) grew. This is the manifest-driven reasoner being either confident enough to walk further (deeper) or terminating immediately at v_root when the entry signature only matches one node (shallower) — fewer "soft 3-hop" guesses in the middle.

Final composition (500 cases)

Source × system

otel-demo slimmed from 64 → 38 because the manifest reasoner only kept 43 of the 64 v2 cases as keepable (the rest hit cyclic / lp≤1); ts and hs filled the slack.

Chaos family

hybrid_clean hit its cap of 150. JVM*/*Stress are notably lighter than v2 (was 35 / 43) because the manifest entry signatures for those families are stricter (jvm.cpu.* / jvm.gc.duration band thresholds) and many pool cases drop to no_paths before reaching layer-1.

Hybrid : leaf split

hybrid:leaf = 217:283 (43.4 % : 56.6 %), inside the [40:60, 60:40] band that v1/v2 used. No ratio enforcement was needed — the natural composition of keepable cases under the new reasoner already produces this split.

longest_path histogram

Selection rules

SYSTEM_CAP = {"ts": 320, "hs": 180, "otel-demo": 150}
FAMILY_CAP = 150
ROOT_CAP   = 30   # any single root service across all families

HARD_FILTERS = (
    "drop cyclic service graphs",
    "drop longest_path <= 1 (single-service blast, no propagation)",
    "drop frontend injections (loadgen / ts-ui-dashboard / frontend / frontend-proxy)",
    "drop missing or malformed injection.json / engine_config",
)
SORT = "(longest_path desc, n_edge desc, n_svc desc, name asc)"

The hybrid:leaf ∈ [40:60, 60:40] ratio guard from v1/v2 is kept as a soft constraint that only kicks in once 200 cases are selected; the two-pass selector relaxes it to fill the last few slots if Pass 1 falls short of 500.

Layout

dataset_v3_500_2026-05-03/
├── manifest.jsonl            # 500 lines, one JSON per case
├── README.md
└── cases/
    └── <case-name>/          # symlink → original case directory
        ├── injection.json     # ground truth (engine_config + display_config)
        ├── causal_graph.json  # manifest-driven service graph (this rebuild)
        ├── env.json
        ├── result.json
        ├── label.txt
        └── *.parquet          # 12 parquets: abnormal_* + normal_*

manifest.jsonl schema per line:

{
  "name": "ts0-ts-order-service-exception-l2bqm5",
  "source": "old|new",
  "system": "ts|hs|otel-demo",
  "longest_path": 7,
  "n_svc": 13,
  "n_edge": 21,
  "n_alarm_svc": 2,
  "root_services": ["ts-order-service"],
  "chaos_family": "JVM*|HTTP*|Network*|Pod*|*Stress|DNS|Time|hybrid_clean|hybrid_kill",
  "primary_kind": "<chaos_type or 'hybrid'>",
  "subtypes": ["<sorted unique chaos_types>"],
  "hybrid": false,
  "has_kill_leg": false,
  "src_path": "/abs/path/to/original/case/dir"
}

Reproducibility

1. Materialize the 1464-case pool (symlink wrap from /dataset/rca/, /detector_success_last13h_2026-05-02/, /detector_success_last48h_gap_2026-05-02/ into _pool_v1_2026-05-02/<case>/converted/, plus a <case>/.valid marker).

2. Run reason batch with manifest-driven reasoning (rcabench-platform commit a130d11 or later, post #370 merge):

   ./cli/reason.py reason batch \
     --data-base-dir _pool_v1_2026-05-02 \
     --max-workers 12 --max-hops 15 --force
   

3. Sync pool/<case>/converted/{causal_graph,result,label,no_*}.* into the original source case dir.

4. Run the curation script (/tmp/curate_v3.py in the build session) with the caps and hard filters listed above.