Reactive AI / Beta Pre Train Corpus Pre training corpus for RxT Beta models, created from public & open datasets. Includes high quality english and polish web crawl data, mathematic and scientific subsets, and code in different programming languages. 2k subsets are filtered for 1024 2048 tokens, except MegaMath Web Pro and GitHub Code subsets, that were filtered for 512 2048 tokens Subsets & original datasets FineWeb Edu fineweb edu s100 (51.3M examples) 50% of 'sample 100BT' subset fineweb edu s100 sh3 1k (18.3M examples) fineweb edu s100 sh3 2k (4.69M examples) third 25% of 'sample 100BT' subset filtered for 1024 2048 tokens examples fineweb edu s100 sh3 4k (2.32M examples) fineweb edu s100 sh4 1k (14.8M examples) fineweb edu s100 sh4 2k (3.90M examples) fourth 25% of 'sample 100BT' subset filtered for 1024 2048 tokens examples fineweb edu s100 sh4 4k (1.86M examples) fineweb edu 2025 08 2k (3.77M examples) CC MAIN 2025 08 subset, filtered for 1024 2048 tokens examples fineweb edu 2025 13 2k (3.95M examples) CC MAIN 2025 13 subset, filtered for 1024 2048 tokens examples fineweb edu 2025 18 1k (11.3M examples) fineweb edu 2025 18 2k (4.13M examples) CC MAIN 2025 18 subset, filtere…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy