Material SciBERT (TPU): Improving language understanding in materials science Work in progress Introduction SciBERT based model pre trained with materials science scientific fulltext Authors Luca Foppiano Pedro Ortiz Suarez TLDR Collected full text from ~700000 articles provided by the National Institute for Materials Science (NIMS) TDM platform (https://dice.nims.go.jp/services/TDM PF/en/), dataset called ScienceCorpus (SciCorpus) We added to the SciBERT vocabulary (32k tokens), 100 domain specific unknown words extracted from SciCorpus with a keywords modeler (KeyBERT) Starting conditions: original SciBERT weights Pre train the model MatTpuSciBERT from on the Google Cloud with the TPU (Tensor Processing Unit) as follow: 800000 steps with batch size: 256, max seq length:512 100000 steps with batch size: 2048, max seq length:128 Fine tuning and testing on NER on superconductors (https://github.com/lfoppiano/grobid superconductors) and physical quantities (https://github.com/kermitt2/grobid quantities) Related work BERT Implementations BERT (the original) https://arxiv.org/abs/1810.04805 RoBERTa (Re implementation by Facebook) https://arxiv.org/abs/1907.11692 Relevant models SciBERT…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy