TrueTeacher This is a Factual Consistency Evaluation model, introduced in the TrueTeacher paper (Gekhman et al, 2023). Model Details The model is optimized for evaluating factual consistency in summarization . It is the main model from the paper (see "T5 11B w. ANLI + TrueTeacher full" in Table 1) which is based on a T5 11B (Raffel et al., 2020) fine tuned with a mixture of the following datasets: TrueTeacher (Gekhman et al., 2023) ANLI (Nie et al., 2020) The TrueTeacher dataset contains model generated summaries of articles from the train split of the CNN/DailyMail dataset (Hermann et al., 2015) which are annotated for factual consistency using FLAN PaLM 540B (Chung et al.,2022). Summaries were generated using summarization models which were trained on the XSum dataset (Narayan et al., 2018). The input format for the model is: "premise: GROUNDING DOCUMENT hypothesis: HYPOTHESIS SUMMARY". To accomodate the input length of common summarization datasets we recommend setting max length to 2048 . The model predicts a binary label ('1' Factualy Consistent, '0' Factualy Inconsistent). Evaluation results This model achieves the following ROC AUC results on the summarization subset of the…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy