USER base U niversal S entence E ncoder for R ussian (USER) is a sentence transformer model for extracting embeddings exclusively for Russian language. It maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search. This model is initialized from deepvk/deberta v1 base and trained to work exclusively with the Russian language. Its quality on other languages was not evaluated. Usage Using this model becomes easy when you have sentence transformers installed: Then you can use the model like this: However, you can use model directly with transformers ⚠️ Attention ⚠️ Each input text should start with "query: " or "passage: ". For tasks other than retrieval, you can simply use the "query: " prefix. Training Details We aimed to follow the bge base en model training algorithm, but we made several improvements along the way. Initialization: deepvk/deberta v1 base First stage: Contrastive pre training with weak supervision on the Russian part of mMarco corpus. Second stage: Supervised fine tuning two different models based on data symmetry and then merging via LM Cocktail : 1. We modified the instruction design by simplify…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy