license: cc-by-nc-4.0
language:
  - en
task_categories:
  - text-generation
  - fill-mask
  - feature-extraction
task_ids:
  - language-modeling
tags:
  - ao3
  - fanfiction
  - creative-writing
  - long-form
  - literature
  - english
  - pretraining
  - chatml
  - openhermes
  - alignment
pretty_name: AO3-2020 Fanfiction Corpus
size_categories:
  - 1B<n<10B
source_datasets:
  - original
multilinguality: monolingual
dataset_size_bytes: 539375493120
configs:
  - name: full
num_examples: 1553596

• name: tokens_2_5M num_examples: 711

• name: tokens_1_25M num_examples: 392

• name: tokens_250K num_examples: 76

• name: tokens_100K num_examples: 25

• name: tokens_75K num_examples: 20

• name: tokens_50K num_examples: 14

• name: tokens_37_5K num_examples: 10

• name: tokens_25K num_examples: 6

• name: tokens_10K num_examples: 2

• name: tokens_7_5K num_examples: 2

• name: tokens_5K num_examples: 1

• name: tokens_2_5K num_examples: 0

• name: three_quarters num_examples: 1165197

• name: seven_tenths num_examples: 1087516

• name: five_ninths num_examples: 863108

• name: half num_examples: 776798

• name: quarter num_examples: 388399

• name: fifth num_examples: 310719

• name: eighth num_examples: 194199

• name: twelfth num_examples: 129466

• name: fifteenth num_examples: 103573

• name: sixteenth num_examples: 97100

• name: twenty_fourth num_examples: 64733

• name: aug_openhermes_10K num_examples: 2

• name: aug_openhermes_37_5K num_examples: 10

• name: aug_openhermes_100K num_examples: 25

• name: aug_openhermes_250K num_examples: 76 dataset_info: features: - name: text dtype: string splits: - name: train num_examples: 15993173 annotations_creators: [] language_creators:

  • crowdsourced

  ---

  AO3-2020 — Archive of Our Own Fanfiction Corpus

  ---

  📋 Overview

  Field	Value
  Total documents	15,993,173
  Total characters	~21,816,825,882
  Estimated tokens	~5,454,206,470
  Raw source size	502 GB (SQLite)
  Language	English
  Source	Archive of Our Own (ao3.org)
  Cutoff date	2020
  License	CC BY-NC 4.0
  Parquet compression	ZSTD level 3
  Last updated	2026-05-17

  ---

  🚀 Quick start

  from datasets import load_dataset
  
  # Full 502 GB corpus — use streaming=True for large machines
  ds = load_dataset("ray0rf1re/AO3-2020", "full", streaming=True)
  
  # Fast experiment — 75 K token slice
  ds = load_dataset("ray0rf1re/AO3-2020", "tokens_75K")
  
  # SFT-ready ChatML format (OpenHermes 2.5 template)
  ds = load_dataset("ray0rf1re/AO3-2020", "aug_openhermes_100K")
  
  # Deterministic 1/8 uniform sample
  ds = load_dataset("ray0rf1re/AO3-2020", "eighth")
  
  # Print first document
  for ex in ds["train"]:
      print(ex["text"][:500])
      break
  

  ---

  🗂️ Subsets

  Full corpus

  Subset	Rows
  full	1,553,596

  Token-capped subsets

  Rows are taken in source order until the cumulative estimated token budget (≈ 4 chars / token) is reached. Good for controlled-scale experiments.

  Subset	Token cap	Rows
  tokens_2_5M	2,500,000	711

| tokens_1_25M | 1,250,000 | 392 | | tokens_250K | 250,000 | 76 | | tokens_100K | 100,000 | 25 | | tokens_75K | 75,000 | 20 | | tokens_50K | 50,000 | 14 | | tokens_37_5K | 37,500 | 10 | | tokens_25K | 25,000 | 6 | | tokens_10K | 10,000 | 2 | | tokens_7_5K | 7,500 | 2 | | tokens_5K | 5,000 | 1 | | tokens_2_5K | 2,500 | 0 |

### Fraction subsets

Deterministic, uniform row-skip sampling — every `d`th row starting at 0.
Reproducible across library versions without a fixed seed.

| Subset | Fraction | Coverage | Rows |
|--------|----------|----------|------|
| `three_quarters` | 3/4 | 75.00 % | 1,165,197 |

| seven_tenths | 7/10 | 70.00 % | 1,087,516 | | five_ninths | 5/9 | 55.56 % | 863,108 | | half | 1/2 | 50.00 % | 776,798 | | quarter | 1/4 | 25.00 % | 388,399 | | fifth | 1/5 | 20.00 % | 310,719 | | eighth | 1/8 | 12.50 % | 194,199 | | twelfth | 1/12 | 8.33 % | 129,466 | | fifteenth | 1/15 | 6.67 % | 103,573 | | sixteenth | 1/16 | 6.25 % | 97,100 | | twenty_fourth | 1/24 | 4.17 % | 64,733 |

### Augmented subsets — OpenHermes 2.5 ChatML template

Same token budgets as their plain counterparts, but each document is
wrapped in the **OpenHermes 2.5 / ChatML** instruction template so the
data can be fed directly to SFT trainers (e.g. LLaMA-Factory, Axolotl,
Unsloth) without additional preprocessing.

Template applied to every row:
```
<|im_start|>system
You are a creative writing assistant specialising in long-form fanfiction. You produce engaging, vivid, and high-quality stories.<|im_end|>
<|im_start|>user
Write a fanfiction story.<|im_end|>
<|im_start|>assistant
{text}<|im_end|>
```

| Subset | Token cap | Rows |
|--------|-----------|------|
| `aug_openhermes_10K` | 10,000 | 2 |

| aug_openhermes_37_5K | 37,500 | 10 | | aug_openhermes_100K | 100,000 | 25 | | aug_openhermes_250K | 250,000 | 76 |

---

## 📁 Data schema

```json
{
    "text":  "string — UTF-8 text of the fanfiction work or chapter.",
    "...":   "All additional metadata columns from the source SQLite are preserved as-is."
}
```

Augmented subsets replace `text` with the ChatML-formatted version.

---



The corpus is directly readable with:
- 🤗 `datasets` (streaming or map-style)
- DuckDB: `SELECT * FROM read_parquet('data/full/*.parquet')`
- Polars: `pl.scan_parquet('data/full/*.parquet')`
- Spark: `spark.read.parquet('data/full/')`
- PyArrow: `pq.read_table('data/full/')`

---

## ⚖️ License & ethics

- **License**: [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/)
- Non-commercial research use only.
- Authors retain copyright over their individual works.
- Works rated **Explicit** or **Mature** may be present in the `full` corpus.
- If you are an AO3 author and wish your work removed, please open an issue.

---

## 🔗 Citation

```bibtex
@dataset{ao3_2020,
  author    = {ray0rf1re},
  title     = {AO3-2020: Archive of Our Own Fanfiction Corpus},
  year      = {2020},
  publisher = {HuggingFace},
  url       = {https://huggingface.co/datasets/ray0rf1re/AO3-2020}
}
```

⚠️ adult_18plus_100K — SAFETY / ALIGNMENT SUBSET

❌ DO NOT train a generative model on this subset as ordinary SFT data.

✅ Use it inverted — as the rejected side of preference pairs.

This subset contains the first ~100,000 tokens of content rated Explicit or Mature on AO3 (26 chapters).

Training on this data naively produces an unsafe model. Training against it — using it as the "what not to do" signal — produces a safer model than one that never saw this kind of content.