CXR BERT specialized CXR BERT is a chest X ray (CXR) domain specific language model that makes use of an improved vocabulary, novel pretraining procedure, weight regularization, and text augmentations. The resulting model demonstrates improved performance on radiology natural language inference, radiology masked language model token prediction, and downstream vision language processing tasks such as zero shot phrase grounding and image classification. First, we pretrain CXR BERT general from a randomly initialized BERT model via Masked Language Modeling (MLM) on abstracts PubMed and clinical notes from the publicly available MIMIC III and MIMIC CXR. In that regard, the general model is expected be applicable for research in clinical domains other than the chest radiology through domain specific fine tuning. CXR BERT specialized is continually pretrained from CXR BERT general to further specialize in the chest X ray domain. At the final stage, CXR BERT is trained in a multi modal contrastive learning framework, similar to the CLIP framework. The latent representation of [CLS] token is utilized to align text/image embeddings. Model variations Model Model identifier on HuggingFace Voc…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy