Multilingual Tokenizer Benchmark

This dataset includes pre-processed wikipedia data for tokenizer evaluation in 45 languages. We provide more information on the evaluation task in general this blogpost.

Usage

The dataset allows us to easily calculate tokenizer fertility and the proportion of continued words on any of the supported languages. In the example below we take the Mistral tokenizer and evaluate its performance on Slovak.

from transformers import AutoTokenizer
from datasets import load_dataset
import numpy as np

def calculate_metrics(tokens):
    tmp = np.array([len(y) for y in tokens])
    return {'fertility': np.mean(tmp), 'cont_prop': np.count_nonzero(tmp > 1) / tmp.shape[0]}

tokenizer_name = 'mistralai/Mistral-7B-v0.1'
language = 'sk' #Slovak
tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)
ds = load_dataset('occiglot/tokenizer-wiki-bench', name=language, split='clean')

remove_columns = list(set(ds.column_names) - set(["text"]))
ds = ds.map(lambda x: {'tokens': tokenizer(x['split_text'], add_special_tokens=False)['input_ids']} ,num_proc=256, remove_columns=remove_columns, batched=False)
remove_columns = None#list(set(ds.column_names))
ds = ds.map(lambda x: calculate_metrics(x['tokens']), num_proc=256, remove_columns=remove_columns,  batched=False)
df = ds.to_pandas()

print('Fertility: ', df.fertility.mean())
print('Prop. continued words:', df.cont_prop.mean())

Dataset Creation

We loosely follow the approach of Rust _et al. using the fast UDPipe to pre-split documents into words and subsequently run the tokenizer over isolated words. For all languages we use the respective November 2023 snapshot from Wikipedia. Since Wikipedia, by nature, contains significantly more numbers and dates than other text and most tokenizers split those into single digits, we filtered all lone-standing numbers from the documents. Additionally, we removed any documents that still contained non-parsed HTML code (less than 1%).

Licensing

We release our curated benchmark and any associated code under MIT license. However, depending on your use case, the licensing conditions of the original Wikipedia data and UDPipe may apply.

Supported Languages

This dataset currently contains pre-processed data for the following languages: