Who Called? V33DA: A Physically Verified Multimodal Benchmark for Vocal Attribution in Zebra Finch Groups

Task. Given a detected zebra finch vocalization and the set of birds visible at that moment, determine which bird produced the call. Caller identity is verified physically from on-body accelerometer vibration; the accelerometer channel is withheld from benchmark models and used only by an oracle ceiling.

V33DA provides 33,625 verified vocalization events from 10 individually identified zebra finches across 3 experiments (2021–2023), each with synchronized 5-channel audio, multi-view video, calibrated 3D pose for every visible candidate, per-bird FM radio telemetry, and accelerometer-derived ground-truth labels.

Code: github.com/marisbasha/v33da  ·  Companion release: songbirdini/v33da_pp (V33DA++: overlap and longer-context buckets)

Quick start

pip install huggingface_hub
huggingface-cli download songbirdini/V33DA --repo-type dataset --local-dir data/v33da

Or in Python:

from huggingface_hub import snapshot_download
snapshot_download("songbirdini/V33DA", repo_type="dataset", local_dir="data/v33da")

Loading a sample

Each binary array column is serialized with numpy.save. To decode:

import io
import numpy as np
import pyarrow.parquet as pq

table = pq.read_table("data/v33da/v33da-00000.parquet")
row = table.to_pydict()

# 3D keypoints: (T_frames, N_birds, 5, 3) float32
kp3d = np.load(io.BytesIO(row["keypoints_3d"][0]))

# Audio path -> multichannel WAV
audio_path = row["audio_path"][0]  # e.g. "audio/juvExpBP01/2021-06-28/..."

The included explore.ipynb notebook shows how to inspect audio, video, pose, radio, and accelerometer data, and how to reproject 3D keypoints into 2D camera views using the shipped calibrations.

Modalities

Each parquet row is one vocalization event:

Ground-truth labels

Labels come from on-body accelerometer vibration, not from microphone-based localization. A WhisperSeg model proposes candidate windows from the demodulated signal; events are retained only when audible in microphones, showing characteristic on-body vibration, with no overlapping call from another bird. All candidates are manually reviewed.

Filtering

From 39,329 reviewed candidates:

• 1,768 rejected as invalid during manual review
• 3,784 removed because another bird vocalized during the same window (overlap filtering)
• 152 removed by 3D-to-2D reprojection error (> 40 px)

Yielding 33,625 released events.

Per-bird counts

Note on bird colors. Backpack color codes are experiment-local identifiers, not global bird IDs. A color that appears in two experiments (e.g. red in BP01 and BP02) refers to two different physical individuals; color-to-bird assignment is fixed within an experiment. All 10 released birds are distinct physical individuals; no bird participates in more than one experiment.

Release contents

Evaluation regimes

The benchmark code supports three evaluation regimes:

Limitations

V33DA is collected in one aviary with one recording geometry across three experiments. Cross-experiment evaluation combines bird-identity shift, group-composition shift, and recording-date shift. The released benchmark excludes overlapping vocalizations and filters out samples with poor geometric consistency. Overlap-filtered calls and longer-context windows are released separately through the companion dataset songbirdini/v33da_pp.

Citation

If you use V33DA, please cite:

@unpublished{basha2026v33da,
  title  = {Who Called? V33DA: A Physically Verified Multimodal Benchmark for Vocal Attribution in Zebra Finch Groups},
  author = {Basha, Maris and Wang, Yuhang and Chen, Xiaoran and Cheng, Longbiao and Yapura, Luca and Zai, Anja T. and Salzmann, Mathieu and Hahnloser, Richard},
  year   = {2026},
  note   = {Under review},
}