FineWeb Tokenized 4 trillion tokens of the pre tokenized data the 🌐 web has to offer Table of Contents What is it? How to download and use Stream a shard with PyArrow Using Hugging Face datasets Dataset Structure Data Format Tokenizer Details Credits & Attribution Citation What is it? This is a pre tokenized version of the HuggingFaceFW/fineweb dataset (currently in progress, tokenization of the ~15 trillion tokens corpus is ongoing). The data is being pre processed and tokenized using the AnisoleAI BPE tokenizer (52,022 vocabulary size) and packed into compact uint16 Parquet shards. By distributing the pre tokenized corpus, we eliminate the CPU and tokenization bottlenecks in pre training pipelines. You can stream these shards directly into your training pipeline without any on the fly tokenization or preprocessing overhead. How to download and use You can load the tokenizer directly from this repository: Stream a shard with PyArrow The most efficient way to use this dataset is to stream the Parquet files and read the token IDs directly as NumPy arrays: Using Hugging Face datasets To load a shard using the standard Hugging Face datasets library: Dataset Structure Data Format Each…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy