Model description The project aims to train sentence embedding models on very large sentence level datasets using a self supervised contrastive learning objective. We used the pretrained 'MiniLM L6 H384 uncased' model and fine tuned in on a 1B sentence pairs dataset. We use a contrastive learning objective: given a sentence from the pair, the model should predict which out of a set of randomly sampled other sentences, was actually paired with it in our dataset. We developped this model during the Community week using JAX/Flax for NLP & CV, organized by Hugging Face. We developped this model as part of the project: Train the Best Sentence Embedding Model Ever with 1B Training Pairs. We benefited from efficient hardware infrastructure to run the project: 7 TPUs v3 8, as well as intervention from Google’s Flax, JAX, and Cloud team member about efficient deep learning frameworks. Intended uses Our model is intented to be used as a sentence encoder. Given an input sentence, it ouptuts a vector which captures the sentence semantic information. The sentence vector may be used for information retrieval, clustering or sentence similarity tasks. How to use Here is how to use this model to ge…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy