Ultra FineWeb L3 📜 Ultra FineWeb Technical Report 📦 UltraData Collection 🌐 UltraData 🤗 MiniCPM5 Series English 中文 📚 Introduction Ultra FineWeb L3 is the L3 refined data for general high quality web data within UltraData's L0 L4 tiered data management framework. Moving beyond L2 quality selection, it transforms high value web corpora into structured, high learnability training data with clearer reasoning signals and richer educational styles. Built on top of Ultra FineWeb, it leverages MiniCPM4 and Qwen3 to perform Q&A Pair Generation and Multi style Rewriting , producing 400B+ English tokens and 200B+ Chinese tokens . To our best knowledge, the Chinese portion is the largest open source Chinese pre training synthetic dataset to date. As high tier data designed for late stage training efficiency, it serves as key training data for the decay phase of MiniCPM5 1B. Ultra FineWeb: Ultra FineWeb, a large scale, high quality, and efficiently filtered dataset , with 1T English tokens and 120B Chinese tokens. Ultra FineWeb classifier: Ultra FineWeb classifier, for filtering high quality data from web corpora. Ultra FineWeb L3: the L3 refined data built on Ultra FineWeb via Q&A Pair Gen…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy