NeoBERT NeoBERT is a next generation encoder model for English text representation, pre trained from scratch on the RefinedWeb dataset. NeoBERT integrates state of the art advancements in architecture, modern data, and optimized pre training methodologies. It is designed for seamless adoption: it serves as a plug and play replacement for existing base models, relies on an optimal depth to width ratio , and leverages an extended context length of 4,096 tokens . Despite its compact 250M parameter footprint, it is the most efficient model of its kind and achieves state of the art results on the massive MTEB benchmark, outperforming BERT large, RoBERTa large, NomicBERT, and ModernBERT under identical fine tuning conditions. Paper: paper Repository: github. Get started Ensure you have the following dependencies installed: If you would like to use sequence packing (un padding), you will need to also install flash attention: How to use Load the model using Hugging Face Transformers: Features Feature NeoBERT Depth to width 28 × 768 Parameter count 250M Activation SwiGLU Positional embeddings RoPE Normalization Pre RMSNorm Data Source RefinedWeb Data Size 2.8 TB Tokenizer google/bert Contex…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy