ModernGBERT 134M ModernGBERT 134M is a German ModernBERT language model with 134 million parameters and a native context length of up to 8,192 tokens. This model follows the same BERT style architecture and training procedure as the ModernBERT codebase. ModernGBERT 134M has been pre trained on 470 billion tokens from the German portion of RedPajama V2, a subset of the same training dataset used for our LLäMmlein decoder family. We provide two model sizes: ModernGBERT 1B 28 layers, hidden size 2,048, 1 billion parameters ModernGBERT 134M ← You are here 22 layers, hidden size 768, 134 million parameters Find more details in our preprint! Usage You can use ModernGBERT with the transformers library from version v4.48.0 onwards. (Optional: install flash attn to achieve highest efficiency.) Since ModernGBERT 134M is a Masked Language Model (MLM), you can load it via AutoModelForMaskedLM . For downstream tasks such as classification, retrieval, or QA, fine tune the model by following standard BERT fine tuning recipes. Example using AutoModelForMaskedLM : NOTE: If you want to use HuggingFace's PEFT library for LoRA training, you need to specify the target modules, e.g.: Intermediate Checkp…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy