SmolLM Corpus: Now shuffled and sharded (and Cleaned)! This is a version of the SmolLM Corpus where the 3 subsets have been interleved, shuffled and sharded as 23698 jsonl.zst files for easy streaming! The dataset is comprised of the cosmopedia v2 and fineweb edu dedup subsets from the original SmolLM Corpus repo, with the python edu subset being pulled from my python edu cleaned repo. Dataset Structure The dataset is split into 24 subdirectories, with the first 23 containing 1000 shards and the 24th containing the final 698. The repository is structured as follows: In general, you can obtain the exact download URL for all shards using the following python function: Generation Code Here is the code which was used to generate the shuffled shards. Note the use of non contiguous interleaving in attempt to uniformly pull documents from across entire subsets to loosely decouple shard index from original document position. Please make sure you pip install zstandard !!! In Memory Decompression Zstandard was chosen as it enables trivial in memory decompression to minimise the storage impact of the dataset. Here is some example code which creates a python generator that yields each json lin…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy