OLMo 2 (November 2024) Pretraining set Collection of data used to train OLMo 2 1124 models. The majority of this dataset comes from DCLM Baseline with no additional filtering, but we provide the explicit breakdowns below. Name Tokens Bytes (uncompressed) Documents License DCLM Baseline 3.70T 21.3TB 2.95B CC BY 4.0 Arxiv 20.8B 77.2GB 3.95M ODC BY pes2o 58.6B 412GB 38M ODC BY starcoder 83.0B 458GB 78.7M ODC BY Algebraic stack 11.8B 44.0GB 2.83M ODC BY OpenWebMath 12.2B 47.23GB 2.89M ODC BY Wiki 3.66B 18.1GB 6.17M ODC BY Total 3.90T 22.4TB 3.08B ODC BY Please refer to the OLMo2 Tech Report for further details. Licensing Information This collection is released under the Open Data Commons Attribution License (ODC By) v1.0 license. The use of this dataset is also subject to CommonCrawl's Terms of Use. Citation A technical manuscript is forthcoming!
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy