Real Robot Manipulation Dataset
7 tabletop manipulation tasks recorded on a UR5e with a Robotiq 2F-140 gripper and two Intel RealSense cameras (one fixed base, one wrist-mounted). 100 episodes per task, 700 episodes total, recorded at 20 Hz.
Tasks
| Folder | Instruction |
|---|---|
put_blocks_in_plate | Pick two blocks from the table and place them on the plate. |
remove_blocks_from_plate | Pick two blocks off the plate and place them on the table. |
hide_blocks_with_bowl | Pair the two blocks together and cover them with the bowl. |
remove_blocks_under_bowl | Uncover the hidden blocks and spread the three objects out. |
hide_blocks_with_cup | Cover each block with an upside-down cup. |
put_blocks_on_cup | Lift each cup off a block and stack the block on top of the cup. |
put_blocks_down | Take the blocks off the cups and place them on the table. |
Each task folder includes a task_instruction.json with the full task spec
(objects, pick/place strategies, gripper modes, workspace bounds).
Layout
<task>/
1.hdf5 ... 100.hdf5 # per-episode trajectories (images + state)
meta/
1/meta.json ... 100/meta.json # per-episode ground truth
task_instruction.json # task spec
HDF5 schema (per episode)
/timestamps (T,) float64
/observations/
images/
camera1 (T,H,W,3) uint8 # base, 640x480 RGB
camera2 (T,H,W,3) uint8 # wrist, 640x480 RGB
qpos (T,7) float32 # 6 joints + gripper (norm)
state (T,14) float32 # tcp_pos(3)+tcp_quat(4)+joints(6)+gripper(1)
tcp_pos, tcp_quat, joints, gripper_pos
meta.json schema (per episode)
For all tasks the meta contains the per-episode ground-truth object positions:
{
"task": "<task name>",
"episode": <int>,
"start": {<objects>: <(x,y)>, ...},
"end": {<objects>: <(x,y)>, ...},
"init_pose": [x, y, z]
}
Task-specific keys are additionally recorded (e.g. perm and cup_order for
the cup tasks, sign and active/passive for the bowl hide task,
block_order for the bowl/cup remove tasks).
Robot setup
- Robot: UR5e via RTDE
- Gripper: Robotiq 2F-140 (auto-calibrated)
- TCP orientation: pointing straight down (axis-angle
[π, 0, 0]) - Cameras: 2× Intel RealSense, 640×480 @ 20 Hz
- Workspace: trapezoid with
y ∈ [0.50, 0.80]andxhalf-width0.10 → 0.18(linearly with y)
Per-task episode counts
| Task | Episodes | Frames (~) | Wall-clock (~) |
|---|---|---|---|
| put_blocks_in_plate | 100 | 45,576 | 38 min |
| remove_blocks_from_plate | 100 | 45,929 | 38 min |
| hide_blocks_with_bowl | 100 | 43,881 | 37 min |
| remove_blocks_under_bowl | 100 | 66,090 | 55 min |
| hide_blocks_with_cup | 100 | (~varies) | ~40 min |
| put_blocks_on_cup | 100 | (~varies) | ~50 min |
| put_blocks_down | 100 | (~varies) | ~30 min |
License
Apache 2.0.