CC-Bench: A Cognitive Conflict Benchmark for MLLMs in Safety-Critical Visual Inspection

CC-Bench is a joint medical-industrial benchmark for evaluating whether multimodal large language models (MLLMs) remain visually grounded when plausible textual context conflicts with image evidence. The benchmark reorganizes public anomaly datasets into a unified four-way multiple-choice QA format for high-risk visual inspection.

This repository currently contains:

• 4,282 images in total
• 2,157 normal images and 2,125 anomalous images
• 11,592 QA instances
• 3 task families: detection, classification, and localization
• 6 cognitive conflict types: Expert, History, Lighting, Machine, Noise, and Time

The benchmark is designed for research on visual grounding, robustness to misleading context, and reliability in safety-critical settings such as medical diagnosis and industrial inspection.

Dataset Summary

Source datasets and benchmark coverage

Benchmark protocol

For each image, CC-Bench provides one or more multiple-choice QA instances that stay anchored to visual evidence:

• Detection: decide whether the image contains a true abnormality
• Classification: identify the fine-grained abnormality type when reliable category semantics are available
• Localization: choose the best normalized bounding box for the abnormal region, or indicate that no reliable lesion/defect is localizable for normal cases

Each QA instance can be evaluated under:

• a clean prompt
• a conflict-conditioned prompt that prepends a misleading textual prior while keeping the image, question, options, and ground-truth answer unchanged

Repository Layout

CC-Bench/
├── data/
│   ├── final_normal.json
│   ├── final_defect.json
│   ├── Br35H/
│   ├── LiverCT/
│   ├── DS-MVTec/
│   └── VisA/
├── evaluation/
│   ├── bias_prompt/
│   ├── examples/
│   ├── results_by_bias_prompt/
│   └── ablation/
├── DATASET_LICENSE.md
├── NEURIPS_RAI.md
└── UPLOAD_CHECKLIST_CN.md

Main files

• data/final_normal.json: benchmark annotations for normal samples
• data/final_defect.json: benchmark annotations for anomalous samples
• evaluation/bias_prompt/*.json: conflict priors used for conflict-conditioned evaluation
• evaluation/examples/: example evaluation scripts for representative MLLMs

If you only need the benchmark annotations, start from data/final_normal.json and data/final_defect.json.

Conflict Types

The repository includes six interpretable misleading context categories:

• Expert: misleading expert or clinician/inspector authority cues
• History: misleading historical statistics or prior records
• Lighting: lighting or exposure explanations that bias interpretation
• Machine: scanner/camera/machine variation explanations
• Noise: sensor noise or contamination explanations
• Time: temporal drift or period-specific explanations

Conflict texts are stored in evaluation/bias_prompt/ and can be paired with clean QA instances for robustness evaluation.

Source Data

CC-Bench is a processed benchmark built from public datasets:

• Br35H
• LiverCT / LiTS-derived data
• DS-MVTec, derived from MVTec AD
• VisA