Bert2Bert Summarization with 🤗 EncoderDecoder Framework This model is a Bert2Bert model fine tuned on summarization. Bert2Bert is a EncoderDecoderModel , meaning that both the encoder and the decoder are bert base uncased BERT models. Leveraging the EncoderDecoderFramework, the two pretrained models can simply be loaded into the framework via: The decoder of an EncoderDecoder model needs cross attention layers and usually makes use of causal masking for auto regressiv generation. Thus, bert2bert is consequently fined tuned on the CNN/Daily Mail dataset and the resulting model bert2bert cnn dailymail fp16 is uploaded here. Example The model is by no means a state of the art model, but nevertheless produces reasonable summarization results. It was mainly fine tuned as a proof of concept for the 🤗 EncoderDecoder Framework. The model can be used as follows: Training script: Please follow this tutorial to see how to warm start a BERT2BERT model: https://colab.research.google.com/drive/1WIk2bxglElfZewOHboPFNj8H44 VAyKE?usp=sharing The obtained results should be: Rouge2 mid precision Rouge2 mid recall Rouge2 mid fmeasure : : : : : : CNN/Daily Mail 16.12 17.07 16.1
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy