Anonymous Review Dataset

This repository contains an evaluation dataset package for anonymous peer review.

Overview

This dataset contains 32,541 evaluation instances across 4 tasks, covering diverse visual domains (animal, plant, food, fungus, artifact, transport, environment, etc.).

Overall Score = mean(MCQ, Sel_Avg, Loc_Avg, Open_QA_corrected)

• Sel_Avg = (Grid-Sel_2x2 + Grid-Sel_3x3) / 2
• Loc_Avg = (Grid-Loc_2x2 + Grid-Loc_3x3) / 2

Directory Structure

.
├── mcq.jsonl
├── sel_2x2.jsonl
├── sel_3x3.jsonl
├── loc_2x2.jsonl
├── loc_3x3.jsonl
├── open_qa.jsonl
└── images/
    ├── single/       (16,095 images for MCQ & Open-QA)
    ├── grid_2x2/     (4,085 images for 2x2 grid tasks)
    └── grid_3x3/     (4,125 images for 3x3 grid tasks)

Data Format

MCQ (mcq.jsonl)

Multiple-choice question with 20 options.

{
  "qid": "mcq_00000003",
  "image": "images/single/000001.jpg",
  "options": ["bellis perennis", "lantana camara", "...", "little golden zinnia", "..."],
  "gt_index": 9,
  "meta": {"domain": "plant", "gt_depth": 4}
}

• options: list of 20 candidate names
• gt_index: index of the correct answer in options

Grid-Sel (sel_2x2.jsonl, sel_3x3.jsonl)

Grid cell selection: identify which cell contains the target object.

{
  "qid": "sel_2x2_001871",
  "image": "images/grid_2x2/000001.jpg",
  "prompt": "You are given a 2x2 grid image. Cells are numbered left-to-right, top-to-bottom as 1, 2, 3, 4. Which cell contains the target category: orange slice? Answer with a single number.",
  "grid_size": "2x2",
  "target_class_name": "orange slice",
  "target_cell": 3,
  "choices": ["1", "2", "3", "4"],
  "answer": "3",
  "meta": {"domain": "food", "image_size": [768, 768], "rows": 2, "cols": 2, "gt_depth": 3}
}

Grid-Loc (loc_2x2.jsonl, loc_3x3.jsonl)

Bounding box prediction: locate the target object with a bounding box.

{
  "qid": "loc_2x2_001871",
  "image": "images/grid_2x2/000001.jpg",
  "prompt": "Locate the target object: orange slice.\n\nOutput the bounding box as [x1, y1, x2, y2]...",
  "grid_size": "2x2",
  "target_class_name": "orange slice",
  "bbox_norm": [0.0, 562.5, 468.5, 937.5],
  "answer": [0.0, 562.5, 468.5, 937.5],
  "meta": {"domain": "food", "image_size": [768, 768], "rows": 2, "cols": 2, "gt_depth": 3}
}

• bbox_norm: ground-truth bounding box [x1, y1, x2, y2] normalized to [0, 1000]
• (0, 0) = top-left, (1000, 1000) = bottom-right

Open-QA (open_qa.jsonl)

Free-form visual identification question.

{
  "qid": "open_qa_00000004",
  "image": "images/single/000001.jpg",
  "question": "What kind of flower is shown in the picture?",
  "gt_aliases_base": ["little golden zinnia", "zinnia grandiflora"],
  "gt_aliases_expanded": ["little golden zinnia", "zinnia grandiflora"],
  "gt_ancestors": [
    {"id": "n12034141", "depth_diff": 1, "names": ["zinnia", "common zinnia"], "score": 0.8},
    {"id": "n11669921", "depth_diff": 2, "names": ["flower"], "score": 0.5}
  ],
  "meta": {"domain": "plant", "gt_depth": 4}
}

• gt_aliases_base / gt_aliases_expanded: acceptable answer strings
• gt_ancestors: hierarchical ancestor nodes for partial-credit scoring

Metadata Fields

• domain: visual category domain (animal, plant, food, fungus, artifact, transport, environment, other, person)
• gt_depth: depth of the ground-truth class in the taxonomy tree (higher = more fine-grained)
• image_size: [width, height] in pixels (grid tasks only)

License

This dataset is released under CC BY-NC 4.0 for research purposes.