Table of Contents:

1. Highlights
2. News
3. Getting Started
4. Benchmark
5. TODO List
6. Contaction

Highlights

1. The SV2V-RSim dataset spans four maps, incorporates four distinct weather conditions and six time periods from sunrise to night, and comprises 203K LiDAR frames, 402K RGB frames, and 788K annotated 3D bounding boxes across 17 object classes. These intricate environments, marked by a high density of road users and constantly evolving traffic patterns, offer a new benchmark dataset for advancing V2V cooperative perception research.
2. Compared with other synthetic datasets, SV2V-RSim provides more realistic rendering quality and more accurate asset geometry.
3. SV2V-RSim supports multiple downstream tasks, including collaborative object detection, depth estimation, semantic segmentation, and more.

News

• [2026.05] 🔥 SV2V-RSim dataset is availale

Getting Started

Data Format

# For SV2V-RSim Dataset
├── train                                                            #training dataset
    ├── {scene_name}                                                 #The scene_name folder contains all agents that have interacted with the ego vehicle.
        ├── {agent_id}		                                           #The agent_id folder contains the sensor annotation information corresponding to the time periods when the agent    
                                                                      interacts with the ego vehicle.
            ├── {timestamp}_{camera_name}_DepthStencil.png           #DepthStencil rendered images, used for subsequent processing into depth information.
            ├── {timestamp}_{camera_name}_ObjectIdentifier.png       #ObjectIdentifier rendered images, used for subsequent processing into segmentation information.
            ├── {timestamp}_{camera_name}_RGB.jpeg                   
            ├── {timestamp}.json                                     #Contains the sensor poses in the world coordinate system, as well as information about all instances placed in the 
                                                                      current scene. The detailed contents are introduced below.
            ├── {timestamp}.pcd                                      #Lidar points
        ├──ego.json                                                  #Ego ID              
├── test                                #testing dataset

# For each JSON file located in {agent_id}/{timestamp}.json. The sensor_visible information is used to compute the occlusion status of each label under the current sensor.
{
  {sensor_name}:{"orientation", "position"},                         #Sensor poses in the world coordinate system
  "type":{type_name},                                                #The category of the current agent
  "labels":{                                                         #Annotation information
    {id}:{                                                           #The unique ID of the annotated object
      "center":[...],                                                #The 3D bounding box center in the world coordinate system
      "size":[...],                                                  #The 3D bounding box size, usually in length, width, and height
      "orientation":[....],                                          #The 3D bounding box orientation in the world coordinate system
      "type":{type_name},                                            #The category of the annotated object
      "name":{asset_name},                                           #The asset name of the annotated object
      "stencil":{stencil_id},                                        #The stencil ID used to identify the object in rendered masks
      "lidar_visibility":{true/false},                               #Whether the object is visible in the lidar point cloud, according to the number of points in the 3d box
      {camera_sensor_name}:                                          #Per-camera visibility information for the annotated object
        {
          "visibility":{true/false},                                 #Whether the object is visible in this camera view
          "visible_pixels":                                          #The number of visible pixels belonging to the object
          "total_pixels":                                            #The total number of pixels belonging to the object before occlusion
          "visible_box_iou":                                         #The IoU between the visible 2D box and the full projected 2D box
        }
    }
  }
}

Data Processing

We provide single-frame examples for decoding depth and semantic annotations. Moreover,

Depth Decoding

depth_extract.py takes one DepthStencil.png image as input and decodes it into metric depth.

python scripts/depth_extract.py \
  --input-depth path_to_xxx_DepthStencil.png \
  --output-dir output_path \
  --format both

The script generates:

• *_depth_m.npy: float32 depth map in meters.
• *_depth_cm.png: 16-bit PNG depth map in centimeters.
• *_valid_mask.png: valid depth mask.
• depth_manifest.csv: statistics for the processed frame.

Semantic Decoding

semantic_extract.py is a single-frame semantic annotation demo. It does not run model inference. Given a frame prefix and camera name, it reads the corresponding RGB.jpeg, ObjectIdentifier.png, and annotation .json files.

python semantic_extract.py \
  --frame-prefix data_path/train/scene/id/timestamp \
  --camera front \
  --output-dir output_path

The script generates:

• *_semantic_id.png: semantic ID map, with pixel values from 0 to 17.
• *_semantic_color.png: colorized semantic segmentation map.
• *_overlay.png: overlay of RGB and semantic segmentation.
• *_RGB.jpeg: copied RGB image.
• *_annotation_info.json: class mapping, object stencil/type/class ID, pixel count, raw labels, and pixel statistics.

OpenCOOD Dataloader for SV2V-RSim

Meanwhile, we provide the data loading file for SV2V-RSim under the OpenCOOD framework: intermediate_fusion_dataset_lv2v.py.

Benchmark

Benchmarking is conducted based on OpenCOOD.

Detection benchmarks based on SV2V-RSim dataset:

TODO List

• Dataset Release
• Data Processing Scripts
• Evaluation Code
• All benchmarks tasks based on SV2V-RSim dataset The remaining parts will be released after the paper is accepted.

Contaction

If any questions and suggenstations, please email to me.