GSA_volc - GSA Embodied Perception Training Dataset
Large-scale Grounding-Spatial-Affordance (GSA) training data for embodied perception Teacher models.
Data Summary
| Metric | Count |
|---|---|
| Total Images | ~580K |
| Total Annotations | 3,267,527 |
| SFT Samples | 1,000,000 |
| GRPO Samples | 250,526 |
| Dimensions | 6 (G_grounding / S_spatial / A_identity_attr / A_obj_state / A_action_precondition / A_action_interface) |
Dimension Distribution (SFT)
| Category | Target | Actual |
|---|---|---|
| G (Grounding) | 40% | 400,000 |
| S (Spatial) | 40% | 400,000 |
| A (Affordance) | 20% | 200,000 |
Directory Structure
''' GSA_volc/ ├── images/ # Images organized by source │ ├── visual_genome/ # 108K images │ ├── grasp_anything/ # 398K images │ ├── benchmark/ # 47K images │ ├── bridgev2/ # 25K images │ └── ... ├── annotations/ │ ├── flywheel_verified/ # Verified by GSA data flywheel (6 dims) │ ├── converted/ # Converted from public datasets (1.6M) │ ├── sft_train.jsonl # ms-swift SFT format (1M) │ └── grpo_train.jsonl # ms-swift GRPO format (250K) ├── flywheel_batches/ # Batch configs for additional generation └── metadata/ ├── manifest.json └── image_entries.jsonl '''
Training Format (SFT - ms-swift compatible)
'''json
{
"messages": [
{"role": "system", "content": "You are an embodied perception assistant..."},
{"role": "user", "content": "\nDetect the cup in this image."},
{"role": "assistant", "content": "{"object": "cup", "bbox": [120, 340, 280, 560]}"}
],
"images": ["images/visual_genome/12345.jpg"]
}
'''
Usage with ms-swift
'''bash export IMAGE_ROOT="/path/to/GSA_volc"
SFT Training
NPROC_PER_NODE=8 swift sft
--model Qwen/Qwen2.5-VL-7B-Instruct
--dataset ${IMAGE_ROOT}/annotations/sft_train.jsonl
--deepspeed zero3 --num_train_epochs 2
--per_device_train_batch_size 1 --max_length 4096
GRPO Training
NPROC_PER_NODE=8 swift rlhf
--rlhf_type grpo
--model Qwen/Qwen2.5-VL-7B-Instruct
--dataset ${IMAGE_ROOT}/annotations/grpo_train.jsonl
--deepspeed zero3
'''
Capability Dimensions
| Dimension | Description | Typical Fields |
|---|---|---|
| G_grounding | Object localization | object_class, bbox |
| S_spatial | Spatial relations | subject, predicate, object, subject_bbox, object_bbox |
| A_identity_attr | Attribute recognition | object_class, attributes |
| A_obj_state | Object state | state_type, state_value |
| A_action_precondition | Action preconditions | can_do_now, blocked_by |
| A_action_interface | Action interface | grasp_point_2d, jaw_axis_2d |
Data Sources
- Visual Genome (108K images): relationships, objects, attributes
- Grasp-Anything (398K images): grasp point annotations
- Bridge Data V2 (25K): tabletop manipulation trajectories
- GSA Data Flywheel (73K verified): auto-generated + verified by VLM pipeline
- Benchmark Eval (47K): robot indoor evaluation scenes
- Additional: SpatialSense, DROID, Embodied-Reasoner, RefSpatial, etc.
Coordinate Convention
All coordinates are normalized to thousandth scale [0, 1000):
- BBox format: [x1, y1, x2, y2] in [0, 999]
- Point format: [x, y] in [0, 999]