Table of contents Introduce Usage Performance Contact Support The Project Citation Introduce ViDense is a VietNamese Embedding Model . Fine tuned and enhanced with tailored methods, ViDense incorporates advanced techniques to optimize performance for text embeddings in various applications. Model Configuration and Methods: Base Model : FacebookAI/xlm roberta large Trained for 10 epochs with a train batch size of 2048. Utilizes a 3 phase training approach, where the best checkpoint from each phase serves as the base model for the next. Position Encoding : Rotary Position Encoding Attention : Blockwise Parallel Transformer Pooling : Mean Pooling Momentum Encoder : Incorporates MoCo (Momentum Contrast) to enhance in batch negative sampling. Rank Encoder : Introduces a Rank Encoder to account for transitive positive relationships. By considering positives of positives as relevant to the anchor, it reranks the corpus using the Spearman metric and integrates Spearman weights into the loss calculation for improved ranking. Loss Function : Cross Entropy Loss (Sim Scores) + MSE Loss (Spearman Scores) Usage Performance Below is a comparision table of the results I achieved compared to some o…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy