Filtered CulturaX + Wikipedia for Dutch This is a combined and filtered version of CulturaX and Wikipedia, only including Dutch. It is intended for the training of LLMs. Different configs are available based on the number of tokens (see a section below with an overview). This can be useful if you want to know exactly how many tokens you have. Great for using as a streaming dataset, too. Tokens are counted as white space tokens, so depending on your tokenizer, you'll likely end up with more tokens than indicated here. Every config also has a test set (for validation) of 1% the total size of the dataset, minimally 1 max. 64k samples (~16M tokens). Wikipedia and CulturaX were shuffled before merging and the test set creation was also shuffled. Priority is given to Wikipedia to prioritize knowledge and cultural content, so the smaller configs will consist exclusively of Wikipedia and for the larger configs we augment with CulturaX. Every config builds further on the previous, so this means that every config contains the same data as the smaller ones and more HOWEVER their train/test splits are not the same, so test set of one config may overlap with samples for another training set. Th…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy