Multilingual Tokenizer Benchmark This dataset includes pre processed wikipedia data for tokenizer evaluation in 45 languages. We provide more information on the evaluation task in general this blogpost. Usage The dataset allows us to easily calculate tokenizer fertility and the proportion of continued words on any of the supported languages. In the example below we take the Mistral tokenizer and evaluate its performance on Slovak. Dataset Creation We loosely follow the approach of Rust et al. using the fast UDPipe to pre split documents into words and subsequently run the tokenizer over isolated words. For all languages we use the respective November 2023 snapshot from Wikipedia. Since Wikipedia, by nature, contains significantly more numbers and dates than other text and most tokenizers split those into single digits, we filtered all lone standing numbers from the documents. Additionally, we removed any documents that still contained non parsed HTML code (less than 1%). Licensing We release our curated benchmark and any associated code under MIT license. However, depending on your use case, the licensing conditions of the original Wikipedia data and UDPipe may apply. Supported Lan…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy