MiniLM: Small and Fast Pre trained Models for Language Understanding and Generation MiniLM is a distilled model from the paper "MiniLM: Deep Self Attention Distillation for Task Agnostic Compression of Pre Trained Transformers". Please find the information about preprocessing, training and full details of the MiniLM in the original MiniLM repository. Please note: This checkpoint uses BertModel with XLMRobertaTokenizer so AutoTokenizer won't work with this checkpoint! Multilingual Pretrained Model Multilingual MiniLMv1 L12 H384: 12 layer, 384 hidden, 12 heads, 21M Transformer parameters, 96M embedding parameters Multilingual MiniLM uses the same tokenizer as XLM R. But the Transformer architecture of our model is the same as BERT. We provide the fine tuning code on XNLI based on huggingface/transformers. Please replace run xnli.py in transformers with ours to fine tune multilingual MiniLM. We evaluate the multilingual MiniLM on cross lingual natural language inference benchmark (XNLI) and cross lingual question answering benchmark (MLQA). Cross Lingual Natural Language Inference XNLI We evaluate our model on cross lingual transfer from English to other languages. Following Conneau e…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy