MultiBLiMP MultiBLiMP is a massively Multilingual Benchmark for Linguistic Minimal Pairs. The dataset is composed of synthetic pairs generated using Universal Dependencies and UniMorph. The paper can be found here. We split the data set by language: each language consists of a single .tsv file. The rows contain many attributes for a particular pair, most important are the sen and wrong sen fields, which we use for evaluating the language models. Using MultiBLiMP To download the final datasets for a given language, you can usethe example code. Make sure to use the ISO 639 3 code, e.g. eng for English (see table below). To run MultiBLiMP on a model, follow the instructions in this repo. Example evaluation code: Languages This table contains the languages covered in MultiBLiMP and the number of items for each language. ISO Code Language n : : : : : : abk Abkhazian 40 aqz Akuntsu 14 sqi Albanian 243 amh Amharic 112 grc Ancient Greek 3695 hbo Ancient Hebrew 983 apu Apurinã 28 hye Armenian 1415 eus Basque 273 bel Belarusian 2570 ben Bengali 21 bho Bhojpuri 34 bor Borôro 241 bre Breton 260 bul Bulgarian 2458 bua Buriat 103 cat Catalan 2284 chu Church Slavonic 4166 xcl Classical Armenian 1…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy