tinyroberta for Extractive QA This is the distilled version of the deepset/roberta base squad2 model. This model has a comparable prediction quality and runs at twice the speed of the base model. Overview Language model: tinyroberta squad2 Language: English Downstream task: Extractive QA Training data: SQuAD 2.0 Eval data: SQuAD 2.0 Code: See an example extractive QA pipeline built with Haystack Infrastructure : 4x Tesla v100 Hyperparameters Distillation This model was distilled using the TinyBERT approach described in this paper and implemented in haystack. Firstly, we have performed intermediate layer distillation with roberta base as the teacher which resulted in deepset/tinyroberta 6l 768d. Secondly, we have performed task specific distillation with deepset/roberta base squad2 as the teacher for further intermediate layer distillation on an augmented version of SQuADv2 and then with deepset/roberta large squad2 as the teacher for prediction layer distillation. Usage In Haystack Haystack is an AI orchestration framework to build customizable, production ready LLM applications. You can use this model in Haystack to do extractive question answering on documents. To load and run th…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy