FineWeb Filtered A filtered subset of HuggingFaceFW/fineweb. Only high quality, clean English web documents are kept. No text is modified — filtering is purely keep/drop. What's in this dataset Each record is a web document with these columns (inherited from FineWeb): Column Description text Full document text id Unique document ID url Source URL dump Common Crawl dump (e.g. CC MAIN 2024 10 ) date Crawl date language Detected language code language score Language ID confidence score token count Estimated token count Filters applied Documents are dropped if they fail any of these checks (in order): Filter Condition to DROP quality Less than 200 characters or 50 words · language score below 0.65 · symbol ratio above 30% · digit ratio above 40% license Matches a blocked license or domain (default: none blocked) ai generated AI probability above 0.85 toxicity Toxicity score above 0.5 perplexity Perplexity outside the normal range (uses entropy proxy if no KenLM model is set) How to use Load with 🤗 Datasets Load a single Parquet file directly Filter by language score in code Source & License Built from HuggingFaceFW/fineweb — licensed ODC By 1.0. This dataset inherits the same license.…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy