Model Card for Switch Transformers Base 8 experts Table of Contents 0. TL;DR 1. Model Details 2. Usage 3. Uses 4. Bias, Risks, and Limitations 5. Training Details 6. Evaluation 7. Environmental Impact 8. Citation 9. Model Card Authors TL;DR Switch Transformers is a Mixture of Experts (MoE) model trained on Masked Language Modeling (MLM) task. The model architecture is similar to the classic T5, but with the Feed Forward layers replaced by the Sparse MLP layers containing "experts" MLP. According to the original paper the model enables faster training (scaling properties) while being better than T5 on fine tuned tasks. As mentioned in the first few lines of the abstract : we advance the current scale of language models by pre training up to trillion parameter models on the “Colossal Clean Crawled Corpus”, and achieve a 4x speedup over the T5 XXL model. Disclaimer : Content from this model card has been written by the Hugging Face team, and parts of it were copy pasted from the original paper. Model Details Model Description Model type: Language model Language(s) (NLP): English License: Apache 2.0 Related Models: All Switch Transformers Checkpoints Original Checkpoints: All Original…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy