LUKE: Deep Contextualized Entity Representations with Entity aware Self attention LUKE ( L anguage U nderstanding with K nowledge based E mbeddings) is a new pre trained contextualized representation of words and entities based on transformer. LUKE treats words and entities in a given text as independent tokens, and outputs contextualized representations of them. LUKE adopts an entity aware self attention mechanism that is an extension of the self attention mechanism of the transformer, and considers the types of tokens (words or entities) when computing attention scores. LUKE achieves state of the art results on five popular NLP benchmarks including SQuAD v1.1 (extractive question answering), CoNLL 2003 (named entity recognition), ReCoRD (cloze style question answering), TACRED (relation classification), and Open Entity (entity typing). Please check the official repository for more details and updates. This is the LUKE base model with 12 hidden layers, 768 hidden size. The total number of parameters in this model is 253M. It is trained using December 2018 version of Wikipedia. Experimental results The experimental results are provided as follows: Task Dataset Metric LUKE large luk…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy