JoyAI-Image-OpenSpatial
Spatial understanding dataset built on OpenSpatial, used in JoyAI-Image.
The full dataset contains about ~3M multi-turn visual-spatial QA samples across 7 open-source datasets and web data. The open-source datasets contain ARKitScenes, ScanNet, ScanNet++, HyperSim, Matterport3D, WildRGB-D, and Ego-Exo4D. Tasks cover a wide range of spatial understanding capabilities including 3D object grounding, depth ordering, spatial relation reasoning, distance estimation, and more. We have released ~2.3M QA samples constructed from the open-source datasets. The remaining web data will be open-sourced in a future release.
Quick Start
from datasets import load_dataset
ds = load_dataset("jdopensource/JoyAI-Image-OpenSpatial", split="train", streaming=True)
for sample in ds:
print(sample["conversations"])
break
Data Format
Each parquet file contains the following columns:
| Column | Type | Description |
|---|---|---|
conversations | list[{from, value}] | Multi-turn conversation pairs (human / gpt). The human turn provides camera parameters and a spatial reasoning question; the gpt turn provides structured spatial annotations (e.g., 3D bounding boxes, depth ordering, spatial relations). |
id | string | Unique sample identifier |
data_source | string | Source dataset (e.g., arkitscenes, scannet, scannetpp, hypersim, matterport3d, wildrgbd, Ego-Exo4D) |
images | list[{bytes, path}] | Embedded image data (PNG bytes) |
type | string | Data type label |
meta_info | string | JSON string with image dimensions (width, height, resized_width, resized_height) |
TODO
- Release 3D lifting data