XLM R Longformer Model / XLM Long XLM R Longformer (or XLM Long for short) is a XLM R model that has been extended to allow sequence lengths up to 4096 tokens, instead of the regular 512. The model was pre trained from the XLM RoBERTa checkpoint using the Longformer pre training scheme on the English WikiText 103 corpus. The reason for this was to investigate methods for creating efficient Transformers for low resource languages, such as Swedish, without the need to pre train them on long context datasets in each respecitve language. The trained model came as a result of a master thesis project at Peltarion and was fine tuned on multilingual quesion answering tasks, with code available here. Since both XLM R model and Longformer models are large models, it it recommended to run the models with NVIDIA Apex (16bit precision), large GPU and several gradient accumulation steps. How to Use The model can be used as expected to fine tune on a downstream task. For instance for QA. Training Procedure The model have been trained on the WikiText 103 corpus, using a 48GB GPU with the following training script and parameters. The model was pre trained for 6000 iterations and took ~5 days. See t…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy