Fineweb Edu Fortified The composition of fineweb edu fortified, produced by automatically clustering a 500k row sample in Airtrain What is it? Fineweb Edu Fortified is a dataset derived from Fineweb Edu by applying exact match deduplication across the whole dataset and producing an embedding for each row. The number of times the text from each row appears is also included as a count column. The embeddings were produced using TaylorAI/bge micro Fineweb and Fineweb Edu were obtained by processing data from 95 crawls of Common Crawl, covering a time period from 2013 to 2024. More information about the original datasets can be found by consulting: Fineweb edu dataset card Fineweb dataset card Fineweb release blog post Fineweb paper The contents of a randomly selected 500k rows from this dataset can be interactively explored in this Airtrain dashboard. Deduplication Deduplication in original Fineweb and Fineweb Edu During creation of the original Fineweb dataset, a variety of deduplication strategies were explored. The evaluation criteria used to assess deduplication strategies was to train ablation models on randomly selected subsets of the data, using a subset of up to ~350 billion to…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy