IGF-Bench: Indoor Geometric Fidelity Benchmark

Anonymous mirror for NeurIPS 2026 Evaluations and Datasets Track double-blind review. The de-anonymised author/maintainer information will replace this header at camera-ready.

IGF-Bench is the first benchmark for evaluating structural-level geometric fidelity of conditionally generated indoor scene images, going beyond perceptual metrics like FID and LPIPS. It pairs 3,600 calibrated synthetic ground-truth views with 21,600 generated images from six state-of-the-art ControlNet models, plus 25,200 monocular-depth estimates, all evaluated with four complementary geometric metrics: planarity (L_plane), orthogonality (L_ortho), edge alignment (L_edge), and vanishing-point consistency (L_vp).

Table of Contents

1. What's in this Repository
2. Headline Findings (verified against this release)
3. Quick Start
4. Directory Structure
5. Reproducing Each Paper Experiment
6. Pre-Computed Result JSONs (no compute needed)
7. Schema Reference
8. License Chain
9. Citation

What's in this Repository

This HuggingFace dataset repository ships everything needed to evaluate or extend IGF-Bench:

• ✅ Ground-truth 3D-FRONT renders with calibrated cameras (RGB + EXR depth + 7-class semantic mask + Canny edge map) at 3 complexity levels (L0_empty / L1_basic / L2_full).
• ✅ Paired generated images from six ControlNet pipelines (SD 1.5, SDXL 1.0, SD 3.5 Large, Flux.1 Dev, Hunyuan-DiT, Kolors) under a uniform protocol (Canny conditioning, seed=42, no negative prompts, cn_scale=1.0).
• ✅ Per-view depth estimates from DepthPro on all 25,200 images, plus DAv2 and ZoeDepth on the 200-view ablation subset (App C.1).
• ✅ Pre-computed evaluation JSONs that already populate every table and figure in the paper (evaluation/).
• ✅ A trained LoRA adapter (experiments/finetune/dav2_lora_adapter/) that demonstrates IGF-Bench is usable as supervision for fine-tuning a pretrained MDE (App C.8).
• ✅ A 200-view MDE consistency subset with DAv2 + ZoeDepth depth pre-computed for paper App C.1 (tab:mde_ablation).
• ✅ A formal Datasheet for Datasets (DATASHEET.md) and Croissant 1.0 metadata (croissant.json).

Headline Findings

Each finding here is reproducible from the JSONs shipped under evaluation/ — no recompute required.

1. Perceptual quality ≠ geometric fidelity. Models with comparable FID (e.g., SDXL FID = 64.2 vs. Flux.1 FID = 56.3) can still differ by ≈2× in ΔL_ortho (Flux.1 0.032 vs. SDXL 0.061). Perceptual metrics alone miss structural failures.
2. Conditioning architecture matters. Flux.1's channel-concatenation conditioning achieves the lowest ΔL_ortho (0.032), substantially better than the residual-injection ControlNet variants used by the other five models.
3. Edge alignment is the universal bottleneck. All six models show 68%–89% relative L_edge degradation versus GT — even the best model fails on fine geometric structure.
4. Cleanliness of synthetic GT. DepthPro AbsRel on IGF-Bench GT is 0.056, vs. 0.084 on NYU-v2 (real Kinect) — the lower MDE-noise floor makes IGF-Bench's relative-degradation (ΔL_*) design more sensitive to generation-induced artefacts.
5. L_ortho works as a label-free quality signal. Picking the per-view best of 5 random Flux.1 seeds by minimum L_ortho cuts L_ortho by 35.1% relative to the best fixed seed (App C.7.1).
6. The synthetic supervision transfers (in-domain). Fine-tuning DepthAnything V2-Small with a 1.75%-parameter LoRA adapter improves in-domain AbsRel from 1.044 to 0.081 (−92%) on the 3D-FRONT held-out test set (App C.8).

Quick Start

Option A — Verify the paper without downloading bulk data

The full evaluation already shipped in evaluation/ is < 50 MB. You can verify every paper number by reading the JSONs directly:

import json
from huggingface_hub import hf_hub_download

# Example: verify the GT L_plane = 0.056 number
fp = hf_hub_download(
    repo_id="igfbench-neurips2026/IGF-Bench",
    filename="evaluation/igf_summary.json",
    repo_type="dataset",
)
data = json.load(open(fp))
gt = next(d for d in data if d["model"] == "GT_baseline")
print(round(gt["l_plane_residual_mean"], 3))   # → 0.056

Option B — Download everything and re-run

pip install huggingface_hub
huggingface-cli download igfbench-neurips2026/IGF-Bench --repo-type dataset \
    --local-dir ./igf-bench-data
export IGF_BENCH_ROOT=$(pwd)/igf-bench-data

Then clone the code at https://anonymous.4open.science/r/IGF-Bench-Code and run:

python scripts/evaluate_igf.py \
    --renders_root  $IGF_BENCH_ROOT/renders_textured \
    --generated_root $IGF_BENCH_ROOT/generated \
    --depth_root    $IGF_BENCH_ROOT/depth_results \
    --output_summary igf_summary.json

This reproduces the full Table 3 from the paper. Runtime ≈ 6 hours on a single RTX 4090; 99% of the time is spent in I/O reading per-view depth NPYs.

Option C — Download only what you need for one row of the main table

For example, to recompute only the Flux.1 row of Table 3 (≈ 70 GB):

huggingface-cli download igfbench-neurips2026/IGF-Bench --repo-type dataset \
    --local-dir ./igf-bench-data \
    --include "renders_textured/*" \
              "generated/flux1_canny/*" \
              "depth_results/gt/depthpro/*" \
              "depth_results/gen/flux1/depthpro/*"

Directory Structure

igfbench-neurips2026/IGF-Bench/
├── README.md                          ← this file
├── DATASHEET.md                       ← formal Datasheets-for-Datasets record
├── croissant.json                     ← MLCommons Croissant 1.0 metadata
├── dataset_card.md                    ← short HF dataset card (4.4 KB summary)
├── LICENSE                            ← CC BY-NC-SA 4.0
├── selected_rooms.json                ← canonical 300-room list
├── room_statistics.json               ← summary statistics
│
├── renders_textured/                  ≈ 36 GB — Ground-truth rendered views
│   └── {scene_id}_{room_type}-{room_id}/
│       └── {L0_empty,L1_basic,L2_full}/
│           └── view_{0,1,2,3}/
│               ├── rgb_textured.png        RGB render
│               ├── depth.exr               metric depth, OpenEXR float32
│               ├── depth.png               8-bit depth visualisation (QA only)
│               ├── depth_gt.npy            NumPy float32 cache of depth.exr (LoRA training I/O)
│               ├── canny.png               Canny edges (thresholds 100/200)
│               ├── semantic_id.png         7-class IDs (0..6)
│               ├── semantic_mask.png       colour-mapped semantic (QA only)
│               ├── wireframe_3d.png        3D wireframe overlay (QA only)
│               └── camera.json             intrinsics + extrinsics
│
├── generated/                         ≈ 36 GB — Generated images per model
│   │
│   │   --- Main protocol: 6 models × 3,600 views = 21,600 PNG (Table 3) ---
│   ├── sd15_canny/                    SD 1.5 native 512² (300 rooms × 3 levels × 4 views)
│   ├── sdxl_canny/                    SDXL 1.0
│   ├── sd35_canny/                    SD 3.5 Large
│   ├── flux1_canny/                   Flux.1 Dev
│   ├── hunyuan_canny/                 Hunyuan-DiT
│   ├── kolors_canny/                  Kolors
│   │
│   │   --- Ablation: negative prompt, App C.3 (3,552 paired views) ---
│   ├── sd15_canny_with_neg/sd15_canny/   SD 1.5 with the legacy negative prompt
│   ├── sdxl_canny_with_neg/sdxl_canny/   SDXL with the legacy negative prompt
│   │
│   │   --- Ablation: resolution, App C.4 (3,552 paired views) ---
│   ├── sd15_upsampled/sd15_canny/     SD 1.5 outputs bicubic-upsampled to 1024²
│   │
│   │   --- Ablation: seed sensitivity (App C.6) + quality gating (App C.7), 200 views each ---
│   └── ablation/
│       ├── flux1_seed123/             App C.6 / C.7 — Flux.1 with seed=123
│       ├── flux1_seed456/             App C.6 / C.7 — Flux.1 with seed=456
│       ├── flux1_seed789/             App C.7 only — Flux.1 with seed=789
│       ├── flux1_seed1024/            App C.7 only — Flux.1 with seed=1024
│       ├── sdxl_w050/                 App C.7 — SDXL with cn_scale=0.50
│       ├── sdxl_w075/                 App C.7 — SDXL with cn_scale=0.75
│       ├── sdxl_w125/                 App C.7 — SDXL with cn_scale=1.25
│       └── sdxl_w150/                 App C.7 — SDXL with cn_scale=1.50
│   (each leaf directory contains the same {room}/{level}/view_{0..3}.png structure)
│
├── depth_results/                     ≈ 150 GB — Per-MDE depth estimates (NPY)
│   ├── gt/                            Ground-truth-render depth (3 MDE backbones)
│   │   ├── depthpro/                  3,600 NPY (full set)
│   │   ├── dav2/                      ablation 200-view subset (App C.1)
│   │   └── zoedepth/                  ablation 200-view subset (App C.1)
│   │
│   └── gen/                           Generated-image depth
│       │
│       │   --- Main protocol: 6 models × 3,600 = 21,600 NPY ---
│       ├── sd15/depthpro/             3,600 NPY
│       ├── sdxl/depthpro/             3,600 NPY  (also dav2/, zoedepth/ for App C.1 200-view subset)
│       ├── sd35/depthpro/             3,600 NPY
│       ├── flux1/depthpro/            3,600 NPY  (also dav2/, zoedepth/ for App C.1 200-view subset)
│       ├── hunyuan/depthpro/          3,600 NPY
│       ├── kolors/depthpro/           3,600 NPY
│       │
│       │   --- Ablation depth maps ---
│       ├── sd15_neg/depthpro/         3,552 NPY  (App C.3 negative prompt)
│       ├── sdxl_neg/depthpro/         3,552 NPY  (App C.3 negative prompt)
│       ├── sd15_upsampled/depthpro/   3,552 NPY  (App C.4 resolution)
│       ├── flux1_seed123/depthpro/    200 NPY    (App C.6 / C.7 seed=123)
│       ├── flux1_seed456/depthpro/    200 NPY    (App C.6 / C.7 seed=456)
│       ├── flux1_seed789/depthpro/    200 NPY    (App C.7 seed=789)
│       ├── flux1_seed1024/depthpro/   200 NPY    (App C.7 seed=1024)
│       ├── sdxl_w050/depthpro/        200 NPY    (App C.7 cn_scale=0.50)
│       ├── sdxl_w075/depthpro/        200 NPY    (App C.7 cn_scale=0.75)
│       ├── sdxl_w125/depthpro/        200 NPY    (App C.7 cn_scale=1.25)
│       └── sdxl_w150/depthpro/        200 NPY    (App C.7 cn_scale=1.50)
│       (each: {room}/{level}/{view}.npy)
│
├── evaluation/                        ≈ 41 MB — Pre-computed metric outputs
│   ├── igf_summary.json               per-model aggregated (Table 3 source)
│   ├── igf_results.json               per-view detailed (≈ 25 MB)
│   ├── error_decomposition.json       App C.2
│   ├── mde_ablation_*.json            App C.1
│   ├── neg_prompt_ablation.json    App C.3
│   ├── n1_resolution_ablation.json    App C.4
│   ├── n3_lvp_improved.json           App C.5
│   ├── seed_ablation.json             App C.6
│   ├── wilcoxon.json               per-pair Wilcoxon + Bonferroni (paper Table 3)
│   ├── anova.json                  two-way Type II ANOVA (paper §4.3)
│   └── fid_lpips.json                 perceptual baselines
│
└── experiments/finetune/
    └── dav2_lora_adapter/             ★ 1.8 MB — Trained LoRA (App C.8)
        ├── adapter_config.json        peft config: r=16, α=32, Q/K/V, dropout=0.05
        ├── adapter_model.safetensors  442,368 trainable params (1.75%)
        └── README.md                  load instructions

Reproducing Each Paper Experiment

Pre-Computed Result JSONs

If you only need to verify the paper's numbers (no recompute), the JSONs below are sufficient.

App F downstream JSONs are released with the paper supplement (https://anonymous.4open.science/r/IGF-Bench-Code) under experiments/results/.

Schema Reference

selected_rooms.json

{
  "rooms": [
    {
      "scene_id": "00110bde-f580-40be-b8bb-88715b338a2a",
      "room_id":  "Bedroom-43072",
      "room_type": "Bedroom",
      "space_type": "Bedroom",
      "area_m2": 12.3
    },
    ...
  ]
}

The on-disk directory name for each room is {scene_id}_{room_id}.

evaluation/igf_summary.json

A list of dicts, one per model and one for GT_baseline:

[
  {
    "model": "GT_baseline",
    "n_samples": 3600,
    "l_plane_residual_mean": 0.056,
    "l_ortho_mean": 0.166,
    "l_edge_f1_mean": 0.606,
    "l_vp_deg_mean": 1.79,
    "blender_abs_rel_mean": 0.0,
    "depthpro_abs_rel_mean": 0.056,
    ...
  },
  {
    "model": "sdxl",
    "n_samples": 3600,
    "delta_l_plane_mean": 0.052,
    "delta_l_ortho_mean": 0.061,
    "delta_l_edge_mean": -0.413,
    ...
  },
  ...
]

evaluation/igf_results.json

A dict keyed by model (gt_baseline, sd15, …, kolors), each holding a list of 3,600 per-view records:

{
  "gt_baseline": [
    {
      "room": "00110bde-..._Bedroom-43072",
      "level": "L0_empty",
      "view": "view_0",
      "l_plane_residual": 0.045,
      "l_plane_inlier_ratio": 0.84,
      "l_ortho": 0.130,
      "n_normals": 5,
      "l_edge_f1": 0.612,
      "l_edge_precision": 0.59,
      "l_edge_recall": 0.64,
      "l_vp_deg": 0.41,
      "depthpro_abs_rel": 0.052,
      "depthpro_rmse": 0.118,
      "depthpro_si_rmse": 0.041,
      "depthpro_delta125": 0.984,
      ...
    },
    ...
  ],
  "sdxl": [...],
  ...
}

experiments/finetune/dav2_lora_adapter/adapter_config.json

{
  "base_model_name_or_path": "depth-anything/Depth-Anything-V2-Metric-Indoor-Small-hf",
  "peft_type": "LORA",
  "r": 16,
  "lora_alpha": 32,
  "lora_dropout": 0.05,
  "target_modules": ["query", "key", "value"],
  "bias": "none"
}

Loadable via peft.PeftModel.from_pretrained(base, adapter_dir).

License Chain

IGF-Bench dataset
├── License: CC BY-NC-SA 4.0
│   • Inherits NC clause from upstream 3D-FRONT (Alibaba Tianchi NC license).
│   • Attribution required, share-alike, non-commercial use only.
│
├── Code (separate): Apache 2.0
│   • Located in the supplementary code repo, not on this dataset HF repo.
│
├── Per-asset upstream model licenses (binding for downstream redistribution):
│   ├── SD 1.5             → CreativeML Open RAIL-M
│   ├── SDXL 1.0           → CreativeML Open RAIL++-M
│   ├── SD 3.5 Large       → Stability AI Community License (NC if revenue ≤ $1M/yr)
│   ├── Flux.1 Dev         → FLUX.1 [dev] Non-Commercial License
│   ├── Hunyuan-DiT        → Tencent Hunyuan Community License
│   ├── Kolors             → Apache 2.0 + Kwai commercial-registration requirement
│   ├── DepthPro           → Apple Sample Code License (apple-amlr; output redistribution
│   │                         in a "gray area" — see DATASHEET §6 for full disclosure)
│   ├── Depth-Anything V2-Small (metric-indoor) → Apache 2.0
│   └── ZoeDepth           → MIT
│
└── Eval-only datasets (NOT redistributed by us; obtain from the official source):
    ├── NYU-v2  → https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html
    └── iBims-1 → https://www.cvg.cit.tum.de/data/datasets/ibims1

For the full per-asset table see DATASHEET.md §6 (Distribution).

The dav2_lora_adapter/ weights inherit CC BY-NC-SA 4.0 from the 3D-FRONT supervision data, even though the base DAv2-S model is Apache 2.0.

Maintenance

The authors commit to maintaining IGF-Bench for at least 5 years post-publication, including:

• Hosting on HuggingFace with versioned releases (v1.0.0, v1.1.0, …);
• Bug fixes via the GitHub Issues tracker (URL pending de-anonymisation);
• Adding new generation models as they become available;
• Periodic re-evaluation when major MDE backbones are released.

Older versions remain accessible on HuggingFace forever (see the refs/convert/parquet/<commit> history).

Citation

@inproceedings{igfbench2026,
  title     = {IGF-Bench: Evaluating Geometric Fidelity of Conditional Image
               Generation Beyond Perceptual Metrics},
  author    = {Anonymous},
  booktitle = {Advances in Neural Information Processing Systems
               (Datasets and Benchmarks Track)},
  year      = {2026}
}

When citing the upstream 3D-FRONT scenes, please also cite Fu et al., 3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics (ICCV 2021).