Model Card for DistilBERT base model (cased) This model is a distilled version of the BERT base model. It was introduced in this paper. The code for the distillation process can be found here. This model is cased: it does make a difference between english and English. All the training details on the pre training, the uses, limitations and potential biases (included below) are the same as for DistilBERT base uncased. We highly encourage to check it if you want to know more. Model description DistilBERT is a transformers model, smaller and faster than BERT, which was pretrained on the same corpus in a self supervised fashion, using the BERT base model as a teacher. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and labels from those texts using the BERT base model. More precisely, it was pretrained with three objectives: Distillation loss: the model was trained to return the same probabilities as the BERT base model. Masked language modeling (MLM): this is part of the original training loss of the BERT base model. When taking a sentence,…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy