Dolma 3 Sample: 150B Mix Dataset Sources Sample of data for 1Bx5C and 7Bx1B. For the full Dolma 3 pool, see: https://huggingface.co/datasets/allenai/dolma3 Source Type Tokens Documents Common Crawl Web pages 121B (76.9%) 84.5M olmOCR Science PDFs Academic documents 19.9B (12.6%) 2.25M Stack Edu (Rebalanced) GitHub code 11.1B (7.06%) 14.3M arXiv Papers with LaTeX 1.29B (0.82%) 247K FineMath 3+ Math web pages 4.10B (2.60%) 2.57M Wikipedia & Wikibooks Encyclopedic 64.6M (0.04%) 119K Total 157B (100%) 104M Licensing Information Dolma 3 is licensed under the Open Data Commons Attribution License v1.0 (ODC By). It is intended for research and educational use. For more information, please see our Responsible Use Guidelines. Citation Find the paper at: https://allenai.org/papers/olmo3
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy