Helsinki-NLP/nemotron-cc-translated
nemotron-cc-tanslated is a collection of automatically translated documents from nemotron-cc taken out of the high-quality subset. Translations are based on OPUS-MT and HPLT-MT models. The data in v1.0 covers 156,431,999 documents with over 70 billion space-searated tokens of English data translated into 36 languages. The total v1.0 data set includes over 2.4 trillion tokens and the translated documents are aligned across all languages.
v1.1 includes a second batch of nemotron-cc-hq data (94 billion space separated tokens) translated into 9 languages: Bulgarian (bul), Czech (ces), Estonian (est), Finnish (fin), Irish (gle), Romanian (ron), Swedish (swe), Turkish (tur) and Ukrainian (ukr).
The translation models used for the translation are the same as for v1.0 (listed below) but translation has been done on CPU using ctranslate2 and beam-search of size 1 instead of using the original Marian-NMT models on GPU with beam search of size 4.
The additional files in v1.1 are marked with -run2- in their file names. Version 1 is marked with -run1-.
More information about how the data has been produced can be found on https://github.com/Helsinki-NLP/translate-fineweb. The v1.0 corpus is also available with aligned sentences in synOPUS: https://opus.nlpl.eu/synthetic/nemotron-cc-translated/v1syn.
Supported Languages
| Langid | Language |
|---|---|
| bos | Bosnian |
| bul | Bulgarian |
| cat | Catalan |
| ces | Czech |
| dan | Danish |
| deu | German |
| ell | Modern Greek |
| eng | English |
| est | Estonian |
| eus | Basque |
| fin | Finnish |
| fra | French |
| gle | Irish |
| glg | Galician |
| hrv | Croatian |
| hun | Hungarian |
| isl | Icelandic |
| ita | Italian |
| kat | Georgian |
| lav | Latvian |
| lit | Lithuanian |
| mkd | Macedonian |
| mlt | Maltese |
| nld | Dutch |
| nno | Norwegian Nynorsk |
| nob | Norwegian Bokmål |
| pol | Polish |
| por | Portuguese |
| ron | Romanian |
| slk | Slovak |
| slv | Slovenian |
| spa | Spanish |
| sqi | Albanian |
| srp_Cyrl | Serbian (cyrillic script) |
| swe | Swedish |
| tur | Turkish |
| ukr | Ukrainian |
Citation Information
Please acknowledge the source when using the data and, please, cite the following article if you use any part of this corpus in your own work:
@article{tiedemann2023democratizing,
title={Democratizing neural machine translation with {OPUS-MT}},
author={Tiedemann, J{\"o}rg and Aulamo, Mikko and Bakshandaeva, Daria and Boggia, Michele and Gr{\"o}nroos, Stig-Arne and Nieminen, Tommi and Raganato, Alessandro and Scherrer, Yves and Vazquez, Raul and Virpioja, Sami},
journal={Language Resources and Evaluation},
number={58},
pages={713--755},
year={2023},
publisher={Springer Nature},
issn={1574-0218},
doi={10.1007/s10579-023-09704-w}
}
Translation Models
We use the original OPUS-MT and HPLT MT models trained with Marian-NMT. The following translation models have been used for creating the data:
Acknowledgements
None of this would be possible without the enormous work done by Common Crawl providing the essential data that most open datasets for language modeling are based on. Furthermore, we are also grateful for the data preparation work done by Nvidia and the community on top of the crawled data from Common Crawl published under the label of nemotron-cc. Important for this work is also the availability of parallel data through OPUS and the public translation models based on that data. Furthermore, this project was supported by the European Union's Horizon Europe research and innovation programme through the HPLT project under grant agreement No 101070350. Finally, we also want to acknowledge the computational resource made available from the Finnish national allocation for the LUMI supercomputer (https://www.lumi-supercomputer.eu) through the extreme scale project MaMuLaM: Massively Multilingual Language Models. None of the translated data would exist without this infrastructure and the compute.