Ultra FineWeb π Technical Report π¦ UltraData Collection π UltraData π€ MiniCPM4 Series π€ MiniCPM5 Series English δΈζ π Introduction Ultra FineWeb is a large scale, high quality, and efficiently filtered dataset . We use the proposed efficient verification based high quality filtering pipeline to the FineWeb and Chinese FineWeb datasets (source data from Chinese FineWeb edu v2, which includes IndustryCorpus2, MiChao, WuDao, SkyPile, WanJuan, ChineseWebText, TeleChat, and CCI3), resulting in the creation of higher quality Ultra FineWeb en with approximately 1T tokens, and Ultra FineWeb zh datasets with approximately 120B tokens, collectively referred to as Ultra FineWeb. Ultra FineWeb serves as a core pre training web dataset for the MiniCPM4 Series and MiniCPM5 Series models. Ultra FineWeb: Ultra FineWeb, a large scale, high quality, and efficiently filtered dataset , with 1T English tokens and 120B Chinese tokens. ( Abstract: Data quality has become a key factor in enhancing model performance with the rapid development of large language models (LLMs). Model driven data filtering has increasingly become a primary approach for acquiring high quality data. However, it still facesβ¦
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy