language: en license: bsd datasets: bookcorpus wikipedia SqueezeBERT pretrained model This model, squeezebert uncased , is a pretrained model for the English language using a masked language modeling (MLM) and Sentence Order Prediction (SOP) objective. SqueezeBERT was introduced in this paper. This model is case insensitive. The model architecture is similar to BERT base, but with the pointwise fully connected layers replaced with grouped convolutions. The authors found that SqueezeBERT is 4.3x faster than bert base uncased on a Google Pixel 3 smartphone. Pretraining Pretraining data BookCorpus, a dataset consisting of thousands of unpublished books English Wikipedia Pretraining procedure The model is pretrained using the Masked Language Model (MLM) and Sentence Order Prediction (SOP) tasks. (Author's note: If you decide to pretrain your own model, and you prefer to train with MLM only, that should work too.) From the SqueezeBERT paper: We pretrain SqueezeBERT from scratch (without distillation) using the LAMB optimizer, and we employ the hyperparameters recommended by the LAMB authors: a global batch size of 8192, a learning rate of 2.5e 3, and a warmup proportion of 0.28. Followi…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy