Dataset description The pile is an 800GB dataset of english text designed by EleutherAI to train large scale language models. The original version of the dataset can be found here. The dataset is divided into 22 smaller high quality datasets. For more information each of them, please refer to the datasheet for the pile. However, the current version of the dataset, available on the Hub, is not splitted accordingly. We had to solve this problem in order to improve the user experience when it comes to deal with the pile via the hub. Here is an instance of the pile We used the meta column to properly divide the dataset in subsets. Each instance example belongs to the subset domain and domain = example['meta']['pile set name'] . By doing this, we were able to create a new version of the pile that is properly divided, each instance having a new column domain . We further splitted each subset in train/test (97%/3%) to build the current dataset which the following structure Usage Using subset of interest = "default" will load the whole dataset.
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy