RIG-bench

Anonymous submission to the NeurIPS 2026 Evaluations & Datasets (E&D) Track.

A benchmark for reasoning-driven image generation: given visual context (images + instruction + optional demonstration pairs), the model must produce the answer as a single image.

• 2,000 samples
• 4 task families × 11 subtasks
• ~1.4 GB

Files

RIG-bench/
├── README.md
├── samples.jsonl              # 2,000 records
└── images/<sample_id>/
    ├── input_<order>.<ext>
    ├── demo_<j>_input_<k>.<ext>, demo_<j>_output_<k>.<ext>   (some samples)
    └── target.<ext>

Data Format

Each line in samples.jsonl corresponds to one benchmark sample. A sample includes the input context, the expected visual answer, task labels.

Loading

import json
from pathlib import Path
from PIL import Image
from huggingface_hub import snapshot_download

ROOT = Path(snapshot_download(repo_id="anonymous-submission-RIG-bench/RIG-Bench",repo_type="dataset"))
samples = [json.loads(l) for l in (ROOT / "samples.jsonl").open(encoding="utf-8")]
s = samples[0]
input_images    = [Image.open(ROOT / img["path"]) for img in s["input"]["images"]]
GT_target_image = Image.open(ROOT / s["output"]["target_image"])