cc12_imagenet21k_recap_hq_bucketed

• Title: cc12_imagenet21k_recap_hq_bucketed
• Description: This ~18M rows dataset is a re upload of https://huggingface.co/datasets/gmongaras/CC12M_and_Imagenet21K_Recap_Highqual where the images have been pre bucketed into SDXL style aspect ratio buckets for target training at ~512^2 and ~256^2 pixels, and where about 7M rows were recaptioned with either Gemini or Ministral. To avoid re encoding the images they have been left untouched so cropping and resizing must be done at loading time

Technical details

This repository contains a bucketed-shards export (uncompressed TAR shards).

Format

• Format: bucketed_shards_v2
• Created: 2026-01-10T15:53:34.486914+00:00
• Export ID: export-2026-01-10T15:53:34.486914+00:00
• Manifest: manifest.json
• Image mode: passthrough_jpeg

Directory layout:

• manifest.json (global metadata + per-bucket shard listing)
• buckets/<bucket_id>/shard-*.tar

Each TAR shard contains 3 files per sample:

• <key>.jpg (JPEG bytes; either re-encoded RGB JPEG or source JPEG passthrough depending on image_mode)
• <key>.txt (caption text, UTF-8, newline-terminated)
• <key>.json (per-sample metadata: w, h, jpeg, image_mode, caption_variant, caption_selector_index, caption_source_id)

Image preprocessing

Unlike other datasets available in this repo, the images have been left unprocesed and are only pre bucketed into target aspect ratio buckets.

Resize/crop intended usage details at load time:

• Cover scale is scale = max(target_w / src_w, target_h / src_h); if scale > 1, the sample is skipped.
• After resize, a crop box is chosen deterministically from the sample key (sha256 of image_id).
  • Corner strategy chooses a corner from allowed_corners where 0=TL, 1=TR, 2=BL, 3=BR (optional small jitter for corner_jitter).

Buckets / resolutions

• Buckets follow SDXL-style proto buckets defined at a 1024×1024 base.
• Base resolution(s): [512, 256]
• In single-res exports, bucket_id is the proto (1024-base) bucket, e.g. p1024x1024.
• In multi-res exports, buckets are namespaced by base resolution: r<base>_<proto>, e.g. r512_p1024x1024.
• The actual target resolution for each bucket (scaled by the per-bucket base resolution and divisible=32) is stored in:
  • manifest.json → buckets[<bucket_id>].scaled.w/h (and base_resolution)
  • each sample’s <key>.json → w/h

Bucket IDs (preview): r256_p1024x1024, r256_p1088x896, r256_p1152x896, r256_p1216x832, r256_p1344x704, r256_p1344x768, r256_p1472x704, r256_p1600x640, r256_p1728x576, r256_p1856x512, r256_p1984x512, r256_p2048x512, r256_p512x1920, r256_p512x2048, r256_p576x1664, r256_p576x1792, r256_p640x1536, r256_p704x1408, r256_p768x1280, r256_p832x1152, … (+42 more)

Bucket distribution:

Caption selection

Available caption variants

Missing caption policy: drop

Export summary

• images_seen: 18,655,051
• images_exported: 18,455,504
• skipped_no_caption: 0
• skipped_too_small: 199,547
• decode_errors: 0
• encode_errors: 0

Efficient loading

Recommended

Treat this as a webdataset-style collection of tar shards:

• Prefer sequential reads of tar files for throughput.
• Shuffle at the shard level (and optionally within-shard) for good randomness without expensive random I/O.
• Use manifest.json to list buckets and shards.

Python (webdataset)

import webdataset as wds

urls = "buckets/*/shard-*.tar"  # glob; adjust if you want a single bucket only
ds = (
    wds.WebDataset(urls)
    .decode("pil")            # decodes .jpg to PIL.Image
    .to_tuple("jpg", "txt", "json")
)
for jpg, caption, meta in ds:
    ...

Python (tarfile, no extra deps)

import io, json, tarfile
from pathlib import Path

tar_path = next(Path("buckets").rglob("shard-*.tar"))
with tarfile.open(tar_path, "r") as tf:
    members = tf.getmembers()
    for m in members:
        if not m.name.endswith(".txt"):
            continue
        key = m.name[:-4]
        caption = tf.extractfile(m).read().decode("utf-8").strip()
        meta = json.loads(tf.extractfile(tf.getmember(key + ".json")).read().decode("utf-8"))
        jpg_bytes = tf.extractfile(tf.getmember(key + ".jpg")).read()
        ...