DistilBert for Dense Passage Retrieval trained with Balanced Topic Aware Sampling (TAS B) We provide a retrieval trained DistilBert based model (we call the dual encoder then dot product scoring architecture BERT Dot) trained with Balanced Topic Aware Sampling on MSMARCO Passage. This instance was trained with a batch size of 256 and can be used to re rank a candidate set or directly for a vector index based dense retrieval . The architecture is a 6 layer DistilBERT, without architecture additions or modifications (we only change the weights during training) to receive a query/passage representation we pool the CLS vector. We use the same BERT layers for both query and passage encoding (yields better results, and lowers memory requirements). If you want to know more about our efficient (can be done on a single consumer GPU in 48 hours) batch composition procedure and dual supervision for dense retrieval training, check out our paper: https://arxiv.org/abs/2104.06967 🎉 For more information and a minimal usage example please visit: https://github.com/sebastian hofstaetter/tas balanced dense retrieval Effectiveness on MSMARCO Passage & TREC DL'19 We trained our model on the MSMARCO s…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy