xmod base X MOD is a multilingual masked language model trained on filtered CommonCrawl data containing 81 languages. It was introduced in the paper Lifting the Curse of Multilinguality by Pre training Modular Transformers (Pfeiffer et al., NAACL 2022) and first released in this repository. Because it has been pre trained with language specific modular components ( language adapters ), X MOD differs from previous multilingual models like XLM R. For fine tuning, the language adapters in each transformer layer are frozen. Usage Tokenizer This model reuses the tokenizer of XLM R. Input Language Because this model uses language adapters, you need to specify the language of your input so that the correct adapter can be activated: A directory of the language adapters in this model is found at the bottom of this model card. Fine tuning In the experiments in the original paper, the embedding layer and the language adapters are frozen during fine tuning. A method for doing this is provided in the code: Cross lingual Transfer After fine tuning, zero shot cross lingual transfer can be tested by activating the language adapter of the target language: Bias, Risks, and Limitations Please refer t…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy