RemBERT (for classification) Pretrained RemBERT model on 110 languages using a masked language modeling (MLM) objective. It was introduced in the paper Rethinking embedding coupling in pre trained language models. A direct export of the model checkpoint was first made available in this repository. This version of the checkpoint is lightweight since it is meant to be finetuned for classification and excludes the output embedding weights. Model description RemBERT's main difference with mBERT is that the input and output embeddings are not tied. Instead, RemBERT uses small input embeddings and larger output embeddings. This makes the model more efficient since the output embeddings are discarded during fine tuning. It is also more accurate, especially when reinvesting the input embeddings' parameters into the core model, as is done on RemBERT. Intended uses & limitations You should fine tune this model for your downstream task. It is meant to be a general purpose model, similar to mBERT. In our paper, we have successfully applied this model to tasks such as classification, question answering, NER, POS tagging. For tasks such as text generation you should look at models like GPT2. Tra…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy