MMLW retrieval roberta large MMLW (muszę mieć lepszą wiadomość) are neural text encoders for Polish. This model is optimized for information retrieval tasks. It can transform queries and passages to 1024 dimensional vectors. The model was developed using a two step procedure: In the first step, it was initialized with Polish RoBERTa checkpoint, and then trained with multilingual knowledge distillation method on a diverse corpus of 60 million Polish English text pairs. We utilised English FlagEmbeddings (BGE) as teacher models for distillation. The second step involved fine tuning the obtained models with contrastrive loss on Polish MS MARCO training split. In order to improve the efficiency of contrastive training, we used large batch sizes 1152 for small, 768 for base, and 288 for large models. Fine tuning was conducted on a cluster of 12 A100 GPUs. ⚠️ 2023 12 26: We have updated the model to a new version with improved results. You can still download the previous version using the v1 tag: AutoModel.from pretrained("sdadas/mmlw retrieval roberta large", revision="v1") ⚠️ Usage (Sentence Transformers) ⚠️ Our dense retrievers require the use of specific prefixes and suffixes when en…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy