BioClinical ModernBERT BioClinical ModernBERT is available in two sizes: base (150M parameters) and large (396M parameters). The model training checkpoints can be found here, and our code is available in our GitHub repository. A step by step guide for reproducing our model training with your own data is available here! Table of Contents 1. Model Summary 2. Usage 3. Training 4. Evaluation 5. License 6. Citation Model Summary BioClinical ModernBERT is a domain adapted encoder that builds on ModernBERT base and large, incorporating long context processing and substantial improvements in speed and performance for biomedical and clinical NLP. BioClinical ModernBERT is trained on the largest biomedical and clinical corpus to date, with over 53.5 billion tokens, and addresses a key limitation of prior clinical encoders by leveraging 20 datasets from diverse institutions, domains, and geographic regions, rather than relying on data from a single source. Usage You can use these models directly with the transformers library starting from v4.48.0: Since BioClinical ModernBERT is a Masked Language Model (MLM), you can use the fill mask pipeline or load it via AutoModelForMaskedLM . To use BioC…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy