DistilCamemBERT =============== We present a distillation version of the well named CamemBERT, a RoBERTa French model version, alias DistilCamemBERT. The aim of distillation is to drastically reduce the complexity of the model while preserving the performances. The proof of concept is shown in the DistilBERT paper and the code used for the training is inspired by the code of DistilBERT. Loss function The training for the distilled model (student model) is designed to be the closest as possible to the original model (teacher model). To perform this the loss function is composed of 3 parts: DistilLoss: a distillation loss which measures the silimarity between the probabilities at the outputs of the student and teacher models with a cross entropy loss on the MLM task ; CosineLoss: a cosine embedding loss. This loss function is applied on the last hidden layers of student and teacher models to guarantee a collinearity between them ; MLMLoss: and finaly a Masked Language Modeling (MLM) task loss to perform the student model with the original task of the teacher model. The final loss function is a combination of these three losses functions. We use the following ponderation: $$Loss = 0.5…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy