I BERT base model This model, ibert roberta base , is an integer only quantized version of RoBERTa, and was introduced in this paper. I BERT stores all parameters with INT8 representation, and carries out the entire inference using integer only arithmetic. In particular, I BERT replaces all floating point operations in the Transformer architectures (e.g., MatMul, GELU, Softmax, and LayerNorm) with closely approximating integer operations. This can result in upto 4x inference speed up as compared to floating point counterpart when tested on an Nvidia T4 GPU. The best model parameters searched via quantization aware finetuning can be then exported (e.g., to TensorRT) for integer only deployment of the model. Finetuning Procedure Finetuning of I BERT consists of 3 stages: (1) Full precision finetuning from the pretrained model on a down stream task, (2) model quantization, and (3) integer only finetuning (i.e., quantization aware training) of the quantized model. Full precision finetuning Full precision finetuning of I BERT is similar to RoBERTa finetuning. For instance, you can run the following command to finetune on the MRPC text classification task. Model Quantization Once you are…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy