T5 Efficient TINY (Deep Narrow version) T5 Efficient TINY is a variation of Google's original T5 following the T5 model architecture. It is a pretrained only checkpoint and was released with the paper Scale Efficiently: Insights from Pre training and Fine tuning Transformers by Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, Donald Metzler . In a nutshell, the paper indicates that a Deep Narrow model architecture is favorable for downstream performance compared to other model architectures of similar parameter count. To quote the paper: We generally recommend a DeepNarrow strategy where the model’s depth is preferentially increased before considering any other forms of uniform scaling across other dimensions. This is largely due to how much depth influences the Pareto frontier as shown in earlier sections of the paper. Specifically, a tall small (deep and narrow) model is generally more efficient compared to the base model. Likewise, a tall base model might also generally more efficient compared to a large model. We generally find that, regardless of size, even if absolute performance might increase…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy