MADLAD 400 Dataset and Introduction MADLAD 400 ( Multilingual Audited Dataset: Low resource And Document level ) is a document level multilingual dataset based on Common Crawl, covering 419 languages in total. This uses all snapshots of CommonCrawl available as of August 1, 2022. The primary advantage of this dataset over similar datasets is that it is more multilingual (419 languages), it is audited and more highly filtered, and it is document level. The main disadvantage is also its strength being more filtered, it may lack the recall needed for some applications. There are two versions released: the noisy dataset, which has no filtering except document level LangID, and the clean dataset, which has a variety of filters applied, though it naturally has a fair amount of noise itself. Each dataset is released in a document level form that has been deduplicated. Loading You can load both the clean and noisy versions of any language by specifing its LangID: ~~~ madlad abt = load dataset("allenai/madlad 400", "abt") ~~~ A list of langagues can also be supplied with a keyword argument: ~~~ madlad multilang = load dataset("allenai/madlad 400", languages=["abt", "ace"]) ~~~ Additionally,…
Runs entirely in your browser via DuckDB-Wasm — this dataset's real data file is loaded once, then queried locally. Nothing is sent to a server.
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy