🌿 iNaturalist Bronze Dataset (Research-Grade, Deduplicated)
Description
This dataset contains research-grade observations from iNaturalist, processed through a bronze-layer pipeline that includes:
- Research-grade only observations (community-verified)
- One photo per observation (deduplicated by observation_uuid, keeping the first photo)
- AVIF encoded images stored as binary in Parquet
- ~5TB total size across over 41,316 shards
Note: This dataset started as all photos, but was tuned to filter downloaded images to only research-grade observations with 1 photo per observation. This prioritizes observation diversity over multiple (often near-identical) photos of the same specimen.
Data Composition
| Metric | Count |
|---|---|
| Total bronze rows (photos) | > 400M |
| Research-grade photos (with images) | > 140M |
| Research-grade observations | > 140M |
| Unique taxa | > 500K |
| Shards | 41,316 Parquet files |
| Photo/Obs count per shard | 10,000 |
| Shard size | ~500MB each |
Top Families
| Family | Photos |
|---|---|
| Asteraceae (daisies/sunflowers) | 11.7M |
| Fabaceae (legumes) | 6.4M |
| Nymphalidae (brush-footed butterflies) | 5.6M |
| Anatidae (ducks/geese) | 4.8M |
| Rosaceae (roses) | 3.6M |
| Apidae (bees) | 3.4M |
| Lamiaceae (mint family) | 3.0M |
| Poaceae (grasses) | 2.9M |
| Accipitridae (hawks/eagles) | 2.9M |
| Orchidaceae (orchids) | 2.7M |
File Structure
data/train/
├── 0000.parquet
├── 0001.parquet
├── ...
└── 41316.parquet.parquet
Schema
| Column | Type | Description |
|---|---|---|
photo_id | int64 | Unique photo identifier |
observation_uuid | string | UUID (deduplicated, one per observation) |
obs_taxon_id | int64 | iNaturalist taxon ID |
taxon_name | string | Scientific name |
ancestry | string | Taxonomic lineage path |
rank | string | species, genus, family, etc. |
image | binary | AVIF encoded image (NULL for non-research-grade) |
license | string | Photo license |
obs_quality_grade | string | "research", "casual", or "needs_id" |
obs_latitude | double | Observation latitude |
obs_longitude | double | Observation longitude |
Note: Only rows with obs_quality_grade == "research" have the image column populated. Other quality grades have NULL images.
Image Format
- Format: AVIF (AV1 Image File Format)
- Quality: 75
- Dimensions: Original (preserved)
- Average size: ~15-30 KB
- Metadata: Stripped (EXIF/XMP removed)
Licenses
Photos include licenses as selected by observers: CC0, CC-BY, CC-BY-NC, CC-BY-SA, CC-BY-NC-SA, CC-BY-ND, CC-BY-NC-ND.
⚠️ Non-commercial research use only
Usage
import polars as pl
# Load only research-grade images
df = pl.read_parquet("data/train/*.parquet")
research_df = df.filter(pl.col("obs_quality_grade") == "research")
# Filter by family
beetles = research_df.filter(pl.col("family") == "Carabidae")
# Access image
row = research_df.first()
image_bytes = row['image']
Processing Pipeline
This dataset was created by:
- Joining photos, observations, and taxonomy
- Writing all photos (research + casual + needs_id) to bronze parquet
- Downloading and converting images only for research-grade observations
- Storing images in the
imagecolumn (NULL for non-research-grade) - Deduplicating to one photo per observation
- Sharding into ~500MB Parquet files
Citation
@misc{bio-lens-bronze,
author = {iNaturalist Contributors},
title = {iNaturalist Bronze Dataset (Research-Grade, Deduplicated)},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/datasets/HirakoSan/bio-lens}}
}
Notes
- The dataset can be further augmented, it would represent over ~12 TB total
- Only ~142.8M rows have images (research-grade)
- The remaining rows (casual/needs_id) have NULL image columns and can be filtered out
- One photo per observation favours observation diversity