Rethinking Video Generation Model for the Embodied World
If you like our project, please give us a star ⭐ on GitHub for the latest update.
Key features
- 4M robotic video clips(10K+ hours) for large-scale video generation training.
- 1300+ fine-grained robotic skills, covering diverse actions and task primitives.
- Multi-modal physical annotations, including RGB, depth, and optical flow.
- Multi-robot and multi-task diversity, spanning various robot types, scenarios, and action skills.
- Rich object interactions, enabling complex and realistic robot behavior modeling.
Dataset Structure
RoVid-X provides structured annotations for each video clip in JSON format, where each entry is indexed by the video filename.
📄 JSON Example
{
"5bebd280785add39d0e40cbea9e0c545.mp4": {
"verb": "grab@@place",
"task_caption": "grab the tape and place it into the basket",
"short_caption": "The robotic arm moves toward the tape, grasps it, and places it into the basket.",
"detailed_caption": "The video opens with an overhead view of a table covered with a red-and-white checkered tablecloth in a room equipped with technical devices. On the table..."
}
}
Download
You can download RoVid-X directly from Hugging Face using the official CLI.
# (Optional) If you are in mainland China, use the Hugging Face mirror
export HF_ENDPOINT=https://hf-mirror.com
# Download the dataset
huggingface-cli download DAGroup-PKU/RoVid-X --local-dir datasets/RoVid-X
📚 Citation
If you find this dataset useful, please cite our paper:
@article{deng2026rethinking,
title={Rethinking Video Generation Model for the Embodied World},
author={Deng, Yufan and Pan, Zilin and Zhang, Hongyu and Li, Xiaojie and Hu, Ruoqing and Ding, Yufei and Zou, Yiming and Zeng, Yan and Zhou, Daquan},
journal={arXiv preprint arXiv:2601.15282},
year={2026}
}