EgoScalerV2 Dataset

This dataset accompanies our work on Developing Vision-Language-Action Model from Egocentric Videos. It provides 6DoF object trajectories paired with egocentric visual observations and natural-language action descriptions, formatted in the LeRobot v2.0 schema so it can be consumed directly by LeRobot-compatible pipelines.

• 🌐 Project page: https://biscue5.github.io/egovla-project-page/
• 📄 Paper: Developing Vision-Language-Action Model from Egocentric Videos (arXiv:2509.21986)
• 🧰 Format: LeRobot v2.0 (Parquet + MP4)
• 🪪 License: Apache-2.0

Dataset Structure

meta/info.json:

{
    "codebase_version": "v2.0",
    "robot_type": "human activity",
    "total_episodes": 45157,
    "total_frames": 1409418,
    "total_tasks": 30214,
    "total_videos": 45157,
    "total_chunks": 46,
    "chunks_size": 1000,
    "fps": 20,
    "splits": {
        "train": "0:45157"
    },
    "data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
    "video_path": "videos/chunk-{episode_chunk:03d}/{video_key}/episode_{episode_index:06d}.mp4",
    "features": {
        "observation.images.cam_high": {
            "dtype": "video",
            "shape": [
                3,
                224,
                224
            ],
            "names": [
                "channel",
                "height",
                "width"
            ],
            "info": {
                "video.fps": 20.0,
                "video.height": 224,
                "video.width": 224,
                "video.channels": 3,
                "video.codec": "av1",
                "video.pix_fmt": "yuv420p",
                "video.is_depth_map": false,
                "has_audio": false
            }
        },
        "observation.state": {
            "dtype": "float32",
            "shape": [
                9
            ],
            "names": {
                "axes": [
                    "x",
                    "y",
                    "z",
                    "r00",
                    "r10",
                    "r20",
                    "r01",
                    "r11",
                    "r21"
                ]
            }
        },
        "action": {
            "dtype": "float32",
            "shape": [
                9
            ],
            "names": {
                "axes": [
                    "x",
                    "y",
                    "z",
                    "r00",
                    "r10",
                    "r20",
                    "r01",
                    "r11",
                    "r21"
                ]
            }
        },
        "original.index": {
            "dtype": "int64",
            "shape": [
                1
            ],
            "names": null
        },
        "timestamp": {
            "dtype": "float32",
            "shape": [
                1
            ],
            "names": null
        },
        "frame_index": {
            "dtype": "int64",
            "shape": [
                1
            ],
            "names": null
        },
        "episode_index": {
            "dtype": "int64",
            "shape": [
                1
            ],
            "names": null
        },
        "index": {
            "dtype": "int64",
            "shape": [
                1
            ],
            "names": null
        },
        "task_index": {
            "dtype": "int64",
            "shape": [
                1
            ],
            "names": null
        }
    }
}

Citation

If you use this dataset, please cite:

@article{yoshida2025developing,
  title   = {Developing Vision-Language-Action Model from Egocentric Videos},
  author  = {Yoshida, Tomoya and Kurita, Shuhei and Nishimura, Taichi and Mori, Shinsuke},
  journal = {arXiv preprint arXiv:2509.21986},
  year    = {2025}
}

The data construction pipeline builds on:

@InProceedings{Yoshida_2025_CVPR,
    author    = {Yoshida, Tomoya and Kurita, Shuhei and Nishimura, Taichi and Mori, Shinsuke},
    title     = {Generating 6DoF Object Manipulation Trajectories from Action Description in Egocentric Vision},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2025},
    pages     = {17370-17382}
}