ClinicalBERT This model card describes the ClinicalBERT model, which was trained on a large multicenter dataset with a large corpus of 1.2B words of diverse diseases we constructed. We then utilized a large scale corpus of EHRs from over 3 million patient records to fine tune the base language model. Pretraining Data The ClinicalBERT model was trained on a large multicenter dataset with a large corpus of 1.2B words of diverse diseases we constructed. Model Pretraining Pretraining Procedures The ClinicalBERT was initialized from BERT. Then the training followed the principle of masked language model, in which given a piece of text, we randomly replace some tokens by MASKs, special tokens for masking, and then require the model to predict the original tokens via contextual text. Pretraining Hyperparameters We used a batch size of 32, a maximum sequence length of 256, and a learning rate of 5e 5 for pre training our models. How to use the model Load the model via the transformers library: Citation Please cite these articles: 1. Wang, G., Liu, X., Ying, Z. et al. Optimized glycemic control of type 2 diabetes with reinforcement learning: a proof of concept trial. Nat Med (2023). https:/…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy