mBART 50 mBART 50 is a multilingual Sequence to Sequence model pre trained using the "Multilingual Denoising Pretraining" objective. It was introduced in Multilingual Translation with Extensible Multilingual Pretraining and Finetuning paper. Model description mBART 50 is a multilingual Sequence to Sequence model. It was introduced to show that multilingual translation models can be created through multilingual fine tuning. Instead of fine tuning on one direction, a pre trained model is fine tuned on many directions simultaneously. mBART 50 is created using the original mBART model and extended to add extra 25 languages to support multilingual machine translation models of 50 languages. The pre training objective is explained below. Multilingual Denoising Pretraining : The model incorporates N languages by concatenating data: D = {D1, ..., DN } where each Di is a collection of monolingual documents in language i . The source documents are noised using two schemes, first randomly shuffling the original sentences' order, and second a novel in filling scheme, where spans of text are replaced with a single mask token. The model is then tasked to reconstruct the original text. 35% of eac…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy