SciNCL SciNCL is a pre trained BERT language model to generate document level embeddings of research papers. It uses the citation graph neighborhood to generate samples for contrastive learning. Prior to the contrastive training, the model is initialized with weights from scibert scivocab uncased. The underlying citation embeddings are trained on the S2ORC citation graph. Paper: Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings (EMNLP 2022 paper). Code: https://github.com/malteos/scincl PubMedNCL: Working with biomedical papers? Try PubMedNCL. How to use the pretrained model Sentence Transformers Transformers Triplet Mining Parameters Setting Value seed 4 triples per query 5 easy positives count 5 easy positives strategy 5 easy positives k 20 25 easy negatives count 3 easy negatives strategy random without knn hard negatives count 2 hard negatives strategy knn hard negatives k 3998 4000 SciDocs Results These model weights are the ones that yielded the best results on SciDocs ( seed=4 ). In the paper we report the SciDocs results as mean over ten seeds. model mag f1 mesh f1 co view map co view ndcg co read map co read ndcg cite map ci…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy