ImageNet-1k WebDataset

This dataset contains ImageNet-1k in WebDataset format (tar files) for efficient streaming.

Dataset Structure

• Training: 129 shards (train-*.tar)
• Validation: 5 shards (validation-*.tar)
• Total size: 147.82 GB

Format

Each tar file contains samples with:

• *.jpg: Image bytes
• *.cls: Label (class ID as text)

Usage

import webdataset as wds

# Training dataset
train_url = "train-{000000..000000000}.tar"
dataset = wds.WebDataset(train_url).shuffle(1000).decode("rgb")

for sample in dataset:
    image = sample["jpg"]  # PIL Image
    label = int(sample["cls"])
    # ... your training code

Source

Converted from ILSVRC/imagenet-1k parquet format.

Generated using MetaDistil ImageNet Pipeline