LongT5 (transient global attention, base sized model) LongT5 model pre trained on English language. The model was introduced in the paper LongT5: Efficient Text To Text Transformer for Long Sequences by Guo et al. and first released in the LongT5 repository. All the model architecture and configuration can be found in Flaxformer repository which uses another Google research project repository T5x. Disclaimer: The team releasing LongT5 did not write a model card for this model so this model card has been written by the Hugging Face team. Model description LongT5 model is an encoder decoder transformer pre trained in a text to text denoising generative setting (Pegasus like generation pre training). LongT5 model is an extension of T5 model, and it enables using one of the two different efficient attention mechanisms (1) Local attention, or (2) Transient Global attention. The usage of attention sparsity patterns allows the model to efficiently handle input sequence. LongT5 is particularly effective when fine tuned for text generation (summarization, question answering) which requires handling long input sequences (up to 16,384 tokens). Intended uses & limitations The model is mostly m…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy