MuRIL: Multilingual Representations for Indian Languages === MuRIL is a BERT model pre trained on 17 Indian languages and their transliterated counterparts. We have released the pre trained model (with the MLM layer intact, enabling masked word predictions) in this repository. We have also released the encoder on TFHub with an additional pre processing module, that processes raw text into the expected input format for the encoder. You can find more details on MuRIL in this paper. Overview This model uses a BERT base architecture [1] pretrained from scratch using the Wikipedia [2], Common Crawl [3], PMINDIA [4] and Dakshina [5] corpora for 17 [6] Indian languages. We use a training paradigm similar to multilingual bert, with a few modifications as listed: We include translation and transliteration segment pairs in training as well. We keep an exponent value of 0.3 and not 0.7 for upsampling, shown to enhance low resource performance. [7] See the Training section for more details. Training The MuRIL model is pre trained on monolingual segments as well as parallel segments as detailed below : Monolingual Data : We make use of publicly available corpora from Wikipedia and Common Crawl…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy