HumanEgo

HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos
🌐 Website  ·  📄 Paper  ·  💻 Code  ·  🎥 Visualization Gallery

Dataset Summary

HumanEgo is a dataset of human egocentric manipulation videos recorded with Project Aria (Gen 1) glasses, paired with Meta MPS annotations (SLAM trajectories + hand tracking) and the full HumanEgo preprocessing output — per-frame RGB, object/arm segmentation, 3D hand & object tracking, and ready-to-train per-frame targets. It supports learning manipulation policies from only minutes of human video (see the paper).

Each recording is a short (~30 s), single-task manipulation clip. The dataset ships both the raw inputs (reproduce everything yourself) and the precomputed outputs (skip the GPU pipeline).

At a Glance

🎥 Browse a visualization of every recording in the HumanEgo Data Gallery.

Dataset Structure

<task>/aria/mps_<task>_<id>_vrs/
├── sample.vrs                 # raw Aria recording (RGB / SLAM / IMU / ...)        [INPUT]
├── slam/                      # MPS SLAM: closed/open-loop trajectory, semidense   [INPUT]
│                              #   points, online calibration, summary
├── hand_tracking/             # MPS hand tracking (hand_tracking_results.csv)      [INPUT]
├── else/                      # VRS metadata + health checks                       [INPUT]
└── preprocess/                # HumanEgo preprocessing output                      [OUTPUT]
    ├── aria_*_analysis.png    # phase / SLAM / hand diagnostic plots
    ├── *_results.json         # cotracker / camtriangulator / kptsselector results
    ├── dinosam_mask_obj*.png  # object segmentation previews
    ├── object_centric.ply/png # final object-centric 3D scene
    ├── vis/                   # visualization clips (aria_vis.mp4, visualkpts_vis.mp4, ...)
    └── all_data.tar           # per-frame data, packed (one tar instead of ~14k files)

all_data.tar unpacks to all_data/<frame>/, with per frame:

training_data.json

{
  "metadata": { "idx", "ts", "w", "h", "fps",
                "k": [3x3 intrinsics], "c2w": [4x4 cam->world], "anchor_key": "obj1" },
  "obs":      { "rgb_path", "mask_arm_path", "mask_obj_path",
                "rgb_WoArm_path", "rgb_WArmObjKpts_path", ... },
  "entities": {
    "hands":   { "right": { "T_hand_to_world": [4x4 SE(3)], "grasp": <float> } },
    "objects": { "obj1": { "T_obj_to_world": [4x4], "is_dynamic": <bool> }, "obj2": { ... } }
  }
}

The hamer / wilor / mediapipe hand-tracking outputs are ablation-only and are not included in this release.

How to Download

No token or login required — the dataset is public.

from huggingface_hub import snapshot_download
# one recording, input only (~0.6 GB) — enough to run preprocessing yourself
snapshot_download("Leo-TX/HumanEgo", repo_type="dataset", local_dir="./data",
    allow_patterns=["serve_bread/aria/mps_serve_bread_000_vrs/*"],
    ignore_patterns=["**/preprocess/*"])

Or use the helper from the code repo (pick task / count):

pip install huggingface_hub
python scripts/download_data.py --task serve_bread --num 20      # first 20, with precomputed output
python scripts/download_data.py --task all --num all             # the whole dataset
python scripts/download_data.py --task serve_bread --input-only  # inputs only, run the pipeline yourself

After downloading, unpack the per-frame archive:

tar -xf <recording>/preprocess/all_data.tar -C <recording>/preprocess

Reproducing the Preprocessing

python -m preprocess.Preprocess \
    --mps_path ./data/serve_bread/aria/mps_serve_bread_000_vrs --task serve_bread

See preprocess/README.md for the full pipeline architecture and a description of every output file, and datacollection/README.md to record and MPS-process your own Aria data.

Dataset Creation

Recordings were captured with Project Aria Gen 1 glasses and processed with Meta's Machine Perception Services (MPS) for SLAM and hand tracking. The HumanEgo preprocessing pipeline then segments each clip into Navigation / Transition / Manipulation phases, isolates the object-centric window, and produces object/arm segmentation (Grounding DINO + SAM 2), 2D keypoint tracking (CoTracker3), 3D triangulation, arm inpainting (LaMa), and the consolidated per-frame training_data.json.

Considerations & Limitations

• Egocentric recordings may contain incidental views of people and environments — use responsibly and in line with Project Aria's terms of use.
• This is a compact, task-focused dataset (2 tasks) designed for few-minutes / zero-shot learning, not a large-scale pretraining corpus.

Citation

@misc{humanego2026,
  title         = {HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos},
  author        = {Wang, Zhi and He, Botao and Yu, Kelin and Lee, Seungjae and Gao, Ruohan and Huang, Furong and Aloimonos, Yiannis},
  year          = {2026},
  eprint        = {2605.24934},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO}
}

License

Released under CC BY-NC 4.0 — free for noncommercial use; commercial use requires a separate license (see the code repo). Built on Project Aria (Gen 1 glasses & MPS) — please also respect their terms of use.

Acknowledgements

This dataset and its preprocessing pipeline build on Project Aria / MPS, CoTracker3, Grounding DINO, SAM 2, LaMa, and Orient-Anything.