Clinical ModernBERT Clinical ModernBERT is a state of the art encoder based transformer tailored specifically for biomedical and clinical text handling context length up to 8192 tokens . Building on the innovations introduced by ModernBERT, this model extends the context window to 8,192 tokens and incorporates domain specific vocabulary refinements. It is designed to produce semantically rich representations that capture both the nuanced syntax of biomedical literature and the intricate semantics of clinical narratives. Usage Pretrained model weights and tokenizer artifacts are provided to facilitate easy integration with your downstream biomedical NLP tasks: Model Overview Below is a table summarizing ModernBERT's key architectural components and their benefits: Feature Description Benefit Extended Context Length Processes sequences up to 8,192 tokens . Captures long range dependencies and full document contexts, essential for complex linguistic tasks. GeGLU Activation Uses the GeGLU activation, a gated variant of GeLU. Enhances non linear representation and model stability by allowing controlled information flow. Rotary Positional Embeddings Implements RoPE to encode relative pos…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy