Reactive AI / Beta Pre-Train Corpus
Pre-training corpus for RxT-Beta models, created from public & open datasets. Includes high-quality english and polish web crawl data, mathematic and scientific subsets, and code in different programming languages.
2k subsets are filtered for 1024-2048 tokens, except MegaMath Web Pro and GitHub Code subsets, that were filtered for 512-2048 tokens
Subsets & original datasets
- FineWeb-Edu
fineweb-edu-s100(51.3M examples) - 50% of 'sample-100BT' subsetfineweb-edu-s100-sh3-1k(18.3M examples)fineweb-edu-s100-sh3-2k(4.69M examples) - third 25% of 'sample-100BT' subset filtered for 1024-2048 tokens examplesfineweb-edu-s100-sh3-4k(2.32M examples)fineweb-edu-s100-sh4-1k(14.8M examples)fineweb-edu-s100-sh4-2k(3.90M examples) - fourth 25% of 'sample-100BT' subset filtered for 1024-2048 tokens examplesfineweb-edu-s100-sh4-4k(1.86M examples)fineweb-edu-2025-08-2k(3.77M examples) - CC-MAIN-2025-08 subset, filtered for 1024-2048 tokens examplesfineweb-edu-2025-13-2k(3.95M examples) - CC-MAIN-2025-13 subset, filtered for 1024-2048 tokens examplesfineweb-edu-2025-18-1k(11.3M examples)fineweb-edu-2025-18-2k(4.13M examples) - CC-MAIN-2025-18 subset, filtered for 1024-2048 tokens examplesfineweb-edu-2025-18-4k(1.88M examples)fineweb-edu-2025-21-1k(10.1M examples)fineweb-edu-2025-21-2k(3.76M examples) - CC-MAIN-2025-21 subset, filtered for 1024-2048 tokens examplesfineweb-edu-2025-21-4k(1.73M examples)fineweb-edu-2025-26(14M examples) - CC-MAIN-2025-26 subset - latest crawl
- FineWiki
finewiki-en(6.6M examples) - english wikipedia subsetfinewiki-pl(1.5M examples) - polish wikipedia subset
- FineWeb
fineweb-s10-2k(1.73M examples) - 'sample-10BT' subset, filtered for 1024-2048 tokens examples
- FineWeb2-HQ
fineweb2-hq-pl(13.3M examples) - high-quality filtered polish web crawl
- FineWeb-2
fineweb2-pl-2k(2.13M examples) - latest 30% of polish subset, filtered for 1024-2048 tokens examplesfineweb2-pl-1k(8M examples)
- DCLM-Edu
dclm-edu-2k(3.59M examples) - examples filtered for 3 and higher educational score and 1024-2048 tokensdclm-edu-s1(1.91M examples)dclm-edu-s2(1.93M examples)dclm-edu-s3(1.90M examples)dclm-edu-s4(1.93M examples)dclm-edu-s5(1.92M examples)dclm-edu-s6(1.93M examples)dclm-edu-s7(1.92M examples)dclm-edu-s8(1.91M examples)dclm-edu-s9(1.92M examples)dclm-edu-s10(1.93M examples)dclm-edu-s11(1.92M examples)dclm-edu-s12(1.91M examples)dclm-edu-s13(1.91M examples)
- FinePdfs-Edu
finepdfs-edu-en-2k(4.38M examples) - english subset filtered for 1024-2048 tokensfinepfds-edu-pl-2k(0.18M examples) - polish subset filtered for 1024-2048 tokens (not used, as it's duplicated in basefinepdfs-pl-2k)finepdfs-edu-en-sh5-0-1k(1.88M examples)finepdfs-edu-en-sh5-1-1k(1.87M examples)finepdfs-edu-en-sh5-2-1k(1.86M examples)finepdfs-edu-en-sh5-3-1k(1.84M examples)finepdfs-edu-en-sh5-4-1k(1.78M examples)
- FinePdfs
finepfds-pl-2k(1.65M examples) - polish subset filtered for 1024-2048 tokensfinepdfs-pl-1k(4.6M examples)finepdfs-pl-4k(1.3M examples)
- FineMath
finemath-4plus(6.7M examples) - full subset with best math qualityinfiwebmath-4plus(6.3M examples) - full subset with best math quality
- MegaMath
megamath-web-pro-2k(5.38M examples) - filtered for 512-2048 tokens MegaMath Web Pro subset
- MegaMath-Web-Pro-Max
megamath-web-pro-max-sh4-0(6.26M examples)megamath-web-pro-max-sh4-1(5.98M examples)megamath-web-pro-max-sh4-2(5.69M examples)megamath-web-pro-max-sh4-3(5.77M examples)
- ProofPile-2
pp2-arxiv(4M examples) - arXiv research paperspp2-algebraic-stack(3.37M examples) - mathematic code examples
- Notebooks:
kaggle(0.58M examples) - from HuggingFaceTB/issues-kaggle-notebooksgithub-jupyter(0.05M examples) - from codeparrot/github-jupyter-code-to-textnotebooks-all(0.63M examples) -kaggleandgithub-jupytercombined
- Code:
beta-code-short(4.51M examples) - filtered and combined code from codeparrot/codeparrot-clean and codeparrot/github-code-cleangithub-code-clean-small-2k(8.17M examples) - filtered for 512-2048 tokens code from loubnabnl/github-code-clean-small- Stack-Edu (~28.15M examples)
stack-edu-c(1.38M examples)stack-edu-cpp(1.9M examples)stack-edu-csharp(1.7M examples)stack-edu-go(0.77M examples)stack-edu-java(3.99M examples)stack-edu-javascript(2.71M examples)stack-edu-markdown(3.28M examples)stack-edu-php(1.67M examples)stack-edu-python(5.6M examples)stack-edu-ruby(0.96M examples)stack-edu-rust(0.51M examples)stack-edu-shell(1.3M examples)stack-edu-sql(0.81M examples)stack-edu-swift(0.63M examples)stack-edu-typescript(0.96M examples)