mmBERT: A Modern Multilingual Encoder TL;DR: A state of the art multilingual encoder trained on 3T+ tokens across 1800+ languages, introducing novel techniques for learning low resource languages during the decay phase. mmBERT is a modern multilingual encoder that significantly outperforms previous generation models like XLM R on classification, embedding, and retrieval tasks. Built on the ModernBERT architecture with novel multilingual training innovations, mmBERT demonstrates that low resource languages can be effectively learned during the decay phase of training. It is also significantly faster than any previous multilingual encoder. Table of Contents Highlights Quick Start Model Description Novel Training Innovations Model Family Training Data Usage Examples Fine tuning Examples Model Architecture Citation Quick Start Installation Usage Model Description mmBERT represents the first significant advancement over XLM R for massively multilingual encoder models. Key features include: 1. Massive Language Coverage Trained on over 1800 languages with progressive inclusion strategy 2. Modern Architecture Built on ModernBERT foundation with Flash Attention 2 and unpadding techniques 3.…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy