Stella PL retrieval mini 8k This is an embedding model based on stella en 400M v5 and further fine tuned for retrieval tasks in Polish. It transforms texts into 1024 dimensional vectors. The model training consisted of two stages: In the first stage, we adapted the model to support the Polish language using the multilingual knowledge distillation method, leveraging a diverse corpus of 20 million Polish English text pairs. The original Stella model and the output of the first stage were limited to a short context of 512 tokens. In the second stage, we extended the context to 8192 tokens and then fine tuned the model using contrastive loss on a dataset comprising 1.5 million queries. Positive and negative passages for each query have been selected with the help of BAAI/bge reranker v2.5 gemma2 lightweight reranker. The model was trained for five epochs with a batch size of 1024 queries. Note: The model uses a custom implementation that requires the XFormers library. For XFormers to function correctly, you need a compatible versions of Flash Attention and PyTorch installed. Before using the model, make sure your XFormers installation is properly configured. Usage (Sentence Transformer…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy