OLMo 2 (November 2024) Pretraining set
Collection of data used to train OLMo-2-1124 models. The majority of this dataset comes from DCLM-Baseline with no additional filtering, but we provide the explicit breakdowns below.
| Name | Tokens | Bytes (uncompressed) | Documents | License |
|---|---|---|---|---|
| DCLM-Baseline | 3.70T | 21.3TB | 2.95B | CC-BY-4.0 |
| Arxiv | 20.8B | 77.2GB | 3.95M | ODC-BY |
| pes2o | 58.6B | 412GB | 38M | ODC-BY |
| starcoder | 83.0B | 458GB | 78.7M | ODC-BY |
| Algebraic-stack | 11.8B | 44.0GB | 2.83M | ODC-BY |
| OpenWebMath | 12.2B | 47.23GB | 2.89M | ODC-BY |
| Wiki | 3.66B | 18.1GB | 6.17M | ODC-BY |
| Total | 3.90T | 22.4TB | 3.08B | ODC-BY |
Please refer to the OLMo2 Tech Report for further details.
Licensing Information
This collection is released under the Open Data Commons Attribution License (ODC-By) v1.0 license. The use of this dataset is also subject to CommonCrawl's Terms of Use.
Citation
A technical manuscript is forthcoming!