ClinicalBERT Bio + Discharge Summary BERT Model The Publicly Available Clinical BERT Embeddings paper contains four unique clinicalBERT models: initialized with BERT Base ( cased L 12 H 768 A 12 ) or BioBERT ( BioBERT Base v1.0 + PubMed 200K + PMC 270K ) & trained on either all MIMIC notes or only discharge summaries. This model card describes the Bio+Discharge Summary BERT model, which was initialized from BioBERT & trained on only discharge summaries from MIMIC. Pretraining Data The Bio Discharge Summary BERT model was trained on all discharge summaries from MIMIC III, a database containing electronic health records from ICU patients at the Beth Israel Hospital in Boston, MA. For more details on MIMIC, see here. All notes from the NOTEEVENTS table were included (~880M words). Model Pretraining Note Preprocessing Each note in MIMIC was first split into sections using a rules based section splitter (e.g. discharge summary notes were split into "History of Present Illness", "Family History", "Brief Hospital Course", etc. sections). Then each section was split into sentences using SciSpacy ( en core sci md tokenizer). Pretraining Procedures The model was trained using code from Googl…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy