NeoDictaBERT: Pushing the Frontier of BERT models in Hebrew This is the base model pretrained only on Hebrew. We recommend using the model pretrained on English and Hebrew here. Following the success of ModernBERT and NeoBERT, we set out to train a Hebrew version of NeoBERT. Introducing NeoDictaBERT : A Next Generation BERT style model trained specifically for Hebrew, technical report coming soon. Supported Context Length: 4,096 (~ 3,200 words) Trained on a total of 235B tokens (5 epochs) with a context length of 1,024, and another 50B tokens with a context length of 4,096. Sample usage: Performance Please see our technical report here for performance metrics. The model outperforms previous SOTA models on almost all benchmarks, with a noticeable jump in the QA scores which indicate a much deeper semantic understanding. Citation If you use NeoDictaBERT in your research, please cite BibTeX: License Shield: [![CC BY 4.0][cc by shield]][cc by] This work is licensed under a [Creative Commons Attribution 4.0 International License][cc by]. [![CC BY 4.0][cc by image]][cc by] [cc by]: http://creativecommons.org/licenses/by/4.0/ [cc by image]: https://i.creativecommons.org/l/by/4.0/88x31.png…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy