BioLinkBERT base BioLinkBERT base model pretrained on PubMed abstracts along with citation link information. It is introduced in the paper LinkBERT: Pretraining Language Models with Document Links (ACL 2022). The code and data are available in this repository. This model achieves state of the art performance on several biomedical NLP benchmarks such as BLURB and MedQA USMLE. Model description LinkBERT is a transformer encoder (BERT like) model pretrained on a large corpus of documents. It is an improvement of BERT that newly captures document links such as hyperlinks and citation links to include knowledge that spans across multiple documents. Specifically, it was pretrained by feeding linked documents into the same language model context, besides a single document. LinkBERT can be used as a drop in replacement for BERT. It achieves better performance for general language understanding tasks (e.g. text classification), and is also particularly effective for knowledge intensive tasks (e.g. question answering) and cross document tasks (e.g. reading comprehension, document retrieval). Intended uses & limitations The model can be used by fine tuning on a downstream task, such as questi…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy