Datacard

This is the official VLABench primitive pretraining dataset converted to the LeRobot format. The dataset contains language-conditioned manipulation trajectories collected with a Franka Panda robot in VLABench simulation.

This LeRobot version is hosted at:

https://huggingface.co/datasets/VLABench/vlabench_primitive_pretrain_lerobot

Source

• Project Page: https://vlabench.github.io/
• Arxiv Paper: https://arxiv.org/abs/2412.18194
• Code: https://github.com/OpenMOSS/VLABench
• Raw primitive dataset: https://huggingface.co/datasets/VLABench/raw_primitive_datasets

Dataset Summary

• Format: LeRobot v2.0
• Robot type: panda
• Split: train
• Episodes: 24,289
• Frames: 2,880,198
• Tasks / language instructions: 275
• FPS: 10
• Videos: 0
• Data chunks: 25
• Data files: 24,289 parquet episode files
• Approximate dataset size: 1.47 TB on Hugging Face

Dataset Structure

The dataset follows the standard LeRobot directory layout:

vlabench_pretrain_primitive
+-- meta
|   +-- info.json
|   +-- tasks.jsonl
|   +-- episodes.jsonl
+-- data
    +-- chunk-000
    |   +-- episode_000000.parquet
    |   +-- episode_000001.parquet
    |   +-- ...
    +-- chunk-001
    |   +-- ...
    +-- chunk-024
        +-- episode_024288.parquet

Episode parquet files are stored with the following path template:

data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet

Metadata files:

• meta/info.json: LeRobot dataset metadata, feature schema, split, FPS, and path templates.
• meta/tasks.jsonl: mapping from task_index to language instruction.
• meta/episodes.jsonl: episode index, task instruction, and episode length.

Features

Each frame contains the following fields:

image          uint8 image, shape (480, 480, 3)
second_image   uint8 image, shape (480, 480, 3)
wrist_image    uint8 image, shape (480, 480, 3)
state          float32, shape (7,)
actions        float32, shape (7,)
timestamp      float32, shape (1,)
frame_index    int64, shape (1,)
episode_index  int64, shape (1,)
index          int64, shape (1,)
task_index     int64, shape (1,)

The three image streams correspond to multi-view RGB observations. state stores the robot state, actions stores the action target, and task_index links each frame to a language instruction in meta/tasks.jsonl.

Example task instructions include:

Please pick the poker 4 of diamonds
Please select the painting of style symbolism.
Put the apple into the plate_seen
Insert the tulip into the vase_seen.
Add ketchup to the dish

Uses

Download the dataset with the Hugging Face CLI:

hf download VLABench/vlabench_primitive_pretrain_lerobot \
  --repo-type dataset \
  --local-dir vlabench_pretrain_primitive

The dataset can then be used with LeRobot-compatible training and evaluation pipelines. The training split covers all episodes:

train: 0:24289

For direct inspection, read meta/info.json for the schema and load individual episode parquet files from data/chunk-*.

Citation

If you use this dataset, please consider to cite:

@article{yin2026two,
  title={Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data},
  author={Yin, Linqi and Zhang, Shiduo and Qiu, Shenling and Li, Chenxin and Fu, Zhaoyang and Xiao, Lei and Wang, Xiang and Yang, Chenchen and Xu, Zhe and Qian, Pengfang and others},
  journal={arXiv preprint arXiv:2606.08520},
  year={2026}
}

@article{zhang2024vlabench,
  title={Vlabench: A large-scale benchmark for language-conditioned robotics manipulation with long-horizon reasoning tasks},
  author={Zhang, Shiduo and Xu, Zhe and Liu, Peiju and Yu, Xiaopeng and Li, Yuan and Gao, Qinghui and Fei, Zhaoye and Yin, Zhangyue and Wu, Zuxuan and Jiang, Yu-Gang and others},
  journal={arXiv preprint arXiv:2412.18194},
  year={2024}
}