ClinicalBERT Bio + Clinical BERT Model The Publicly Available Clinical BERT Embeddings paper contains four unique clinicalBERT models: initialized with BERT Base ( cased L 12 H 768 A 12 ) or BioBERT ( BioBERT Base v1.0 + PubMed 200K + PMC 270K ) & trained on either all MIMIC notes or only discharge summaries. This model card describes the Bio+Clinical BERT model, which was initialized from BioBERT & trained on all MIMIC notes. Pretraining Data The Bio ClinicalBERT model was trained on all notes from MIMIC III, a database containing electronic health records from ICU patients at the Beth Israel Hospital in Boston, MA. For more details on MIMIC, see here. All notes from the NOTEEVENTS table were included (~880M words). Model Pretraining Note Preprocessing Each note in MIMIC was first split into sections using a rules based section splitter (e.g. discharge summary notes were split into "History of Present Illness", "Family History", "Brief Hospital Course", etc. sections). Then each section was split into sentences using SciSpacy ( en core sci md tokenizer). Pretraining Procedures The model was trained using code from Google's BERT repository on a GeForce GTX TITAN X 12 GB GPU. Model…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy