A french sequence to sequence pretrained model based on BART. BARThez is pretrained by learning to reconstruct a corrupted input sentence. A corpus of 66GB of french raw text is used to carry out the pretraining. Unlike already existing BERT based French language models such as CamemBERT and FlauBERT, BARThez is particularly well suited for generative tasks (such as abstractive summarization), since not only its encoder but also its decoder is pretrained. In addition to BARThez that is pretrained from scratch, we continue the pretraining of a multilingual BART mBART which boosted its performance in both discriminative and generative tasks. We call the french adapted version mBARThez. Model Architecture layers params : : : : : : BARThez BASE 12 165M mBARThez LARGE 24 458M paper: https://arxiv.org/abs/2010.12321 \ github: https://github.com/moussaKam/BARThez
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy