CulturaY

CulturaY: A Large Cleaned Multilingual Dataset of 75 Languages

Dataset Summary

From the team that brought you CulturaX, we present CulturaY, another substantial multilingual dataset of 15TB (uncompressed)/3TB (zstd-compressed) that applies the same dataset cleaning methodology to the HPLT v1.1 dataset. Please note that HPLT v1.2 has also been released and is an alternative verison with different cleaning methodolgies. This data was used in part to train our SOTA Vietnamese model: Vistral-7B-Chat.

Our annotations and arrangements are licensed under CC-BY-4.0, and we make the data available for fair use machine learning research.
But we make no claims as to the underlying copyrights of the work. This data was copied from the HPLT project, which in turn used the data from Common Crawl and the Internet Archive.

Acknowledgement

We thank our collaborators at UONLP - The Natural Language Processing Group at the University of Oregon, and the computing resources of the managers of the Karolina Supercomputers. We also thank our friends at TurkuNLP for their support.

Data Breakdown:

There are 75 langauges, with the following breakdown:

Dataset structure

The dataset has a total of 6 columns, including:

• 2 columns text, url will be the two main columns in this dataset.
• the remaining columns id, document_lang, scores, langs belong to the original document in the HPLT V1.1 dataset, retained for debugging purposes. and will be removed in the future.

Therefore, when using, please only utilize the two columns text and url.

Process for Creating CulturaY

Firstly, to create CulturaY, we began with the HPLT dataset (version 1.1). This is also a notable difference between X and Y. While X was generated from cleaning data from Common Crawl (mC4, Oscar), Y was generated from cleaning raw data from the Internet Archive (HPLT). While Common Crawl is quite popular, data from the Internet Archive is less known and exploited, even though the data from both sources are similar. HPLT or CulturaY could be considered the first publicly released datasets originating from the Internet Archive. Using both CulturaX and CulturaY simultaneously will help your model have a more diverse source of data.

Our pipeline is built based on Bloom's data cleaning pipeline: evaluating each document in the dataset according to criteria such as document length, perplexity, bad words ratio, etc., and removing documents that do not perform well in any of these criteria. See our Blog for more details.

Citation

To cite CulturaY, please use:

@misc{nguyen2024culturay,
      title={CulturaY: A Large Cleaned Multilingual Dataset of 75 Languages}, 
      author={Thuat Nguyen, Huu Nguyen and Thien Nguyen},
      year={2024},
}