ModernBERT Table of Contents 1. Model Summary 2. Usage 3. Evaluation 4. Limitations 5. Training 6. License 7. Citation Model Summary ModernBERT is a modernized bidirectional encoder only Transformer model (BERT style) pre trained on 2 trillion tokens of English and code data with a native context length of up to 8,192 tokens. ModernBERT leverages recent architectural improvements such as: Rotary Positional Embeddings (RoPE) for long context support. Local Global Alternating Attention for efficiency on long inputs. Unpadding and Flash Attention for efficient inference. ModernBERT’s native long context length makes it ideal for tasks that require processing long documents, such as retrieval, classification, and semantic search within large corpora. The model was trained on a large corpus of text and code, making it suitable for a wide range of downstream tasks, including code retrieval and hybrid (text + code) semantic search. It is available in the following sizes: ModernBERT base 22 layers, 149 million parameters ModernBERT large 28 layers, 395 million parameters For more information about ModernBERT, we recommend our release blog post for a high level overview, and our arXiv pre p…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy