!! Note: this dataset is currently being uploaded and processed. The .bin files are intermediate files to allow shuffling. !! DCLM Baseline Pretokenized (LLaMA 3.1, 8192 context) This dataset is a pretokenized and globally shuffled version of DCLM Baseline (mlfoundations/dclm baseline 1.0), prepared for large scale language model pretraining. It is intended to be used as a direct drop in pretraining corpus for LLaMA 3.1 style training pipelines. The original DCLM Baseline dataset is a 4T token / 3B document pretraining dataset that achieves strong performance on language model benchmarks. See the original dataset card for benchmark comparisons and further details about the source data. Tokenization Every document from DCLM Baseline was tokenized using the LLaMA 3.1 tokenizer: Tokenizer: meta llama/Llama 3.1 8B Library: HuggingFace transformers, AutoTokenizer Each document is independently encoded via tokenizer.encode(document) No padding, no truncation at the tokenization stage Total documents: 2,949,254,346 Approximate total tokens: ~4T Global Shuffle A global document level shuffle was applied across all 2,949,254,346 documents using a fixed random seed for full reproducibility:…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy