Google's T5 Version 1.1 Version 1.1 T5 Version 1.1 includes the following improvements compared to the original T5 model GEGLU activation in feed forward hidden layer, rather than ReLU see here. Dropout was turned off in pre training (quality win). Dropout should be re enabled during fine tuning. Pre trained on C4 only without mixing in the downstream tasks. no parameter sharing between embedding and classifier layer "xl" and "xxl" replace "3B" and "11B". The model shapes are a bit different larger d model and smaller num heads and d ff . Note : T5 Version 1.1 was only pre trained on C4 excluding any supervised training. Therefore, this model has to be fine tuned before it is useable on a downstream task. Pretraining Dataset: C4 Other Community Checkpoints: here Paper: Exploring the Limits of Transfer Learning with a Unified Text to Text Transformer Authors: Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu Abstract Transfer learning, where a model is first pre trained on a data rich task before being fine tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The e…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy