Multilingual E5 base Multilingual E5 Text Embeddings: A Technical Report. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei, arXiv 2024 This model has 12 layers and the embedding size is 768. Usage Below is an example to encode queries and passages from the MS MARCO passage ranking dataset. Supported Languages This model is initialized from xlm roberta base and continually trained on a mixture of multilingual datasets. It supports 100 languages from xlm roberta, but low resource languages may see performance degradation. Training Details Initialization : xlm roberta base First stage : contrastive pre training with weak supervision Dataset Weak supervision of text pairs Filtered mC4 (title, page content) 1B CC News (title, news content) 400M NLLB translation pairs 2.4B Wikipedia (hierarchical section title, passage) 150M Filtered Reddit (comment, response) 800M S2ORC (title, abstract) and citation pairs 100M Stackexchange (question, answer) 50M xP3 (input prompt, response) 80M Miscellaneous unsupervised SBERT data 10M Second stage : supervised fine tuning Dataset Language of text pairs MS MARCO English 500k NQ English 70k Trivia QA English 60k NLI from Sim…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy