MonTok: A Suite of Monolingual Tokenizers This is a set of monolingual tokenizers for 98 languages. For each language, there are Unigram, BPE, and SuperBPE tokenizers, ranging in vocabulary size from around 6k to over 200k. Training Details Training Data All tokenizers are trained on samples of the data used to the train the Goldfish language models. The tokenizers were either trained on scaled or unscaled data. This refers to whether the models are trained on byte premium scaled data or not. We used the Byte Premium Tool to calculate byte premiums. The dataset size is 300MB of data. For unscaled data tokenizers, this means they were trained on 300MB of data. For scaled data tokenizers, the tokenizers were trained on 300 times the byte premium MB of data. Training The BPE and Unigram tokenizers were trained using the Hugging Face implementations and the trainers in the tokenizers package. BPE tokenizers were trained on a range of vocabularies: 6144, 8192, 16384, 32768, 49152, 65536, 81920, 98304, 114688, 262144. Unigram tokenizers were trained on a range of vocab sizes: 8192, 16384, 32768, 49152, 65536, 81920, 98304, 114688; however, some of the smaller vocabulary sizes were too sm…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy