📀 Falcon RefinedWeb Falcon RefinedWeb is a massive English web dataset built by TII and released under an ODC By 1.0 license. See the 📓 paper on arXiv for more details. RefinedWeb is built through stringent filtering and large scale deduplication of CommonCrawl; we found models trained on RefinedWeb to achieve performance in line or better than models trained on curated datasets, while only relying on web data. RefinedWeb is also "multimodal friendly": it contains links and alt texts for images in processed samples. This public extract should contain 500 650GT depending on the tokenizer you use, and can be enhanced with the curated corpora of your choosing. This public extract is about ~500GB to download, requiring 2.8TB of local storage once unpacked. RefinedWeb is the main dataset we have used for training the Falcon LLM models: It was used in conjunction with a curated corpora to train Falcon 7B/40B, two state of the art open source models. It was also used to train Falcon RW 1B/7B, two models trained on 350 billion tokens of RefinedWeb alone to demonstrate its quality compared to curated corpora. Dataset card for Falcon RefinedWeb Dataset Description Homepage: falconllm.tii.a…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy