This repository is primarily for the Transformers framework. If you're using other open source frameworks, please use the alternative repository: MiniMax Text 01 WeChat MiniMax Text 01 1. Introduction MiniMax Text 01 is a powerful language model with 456 billion total parameters, of which 45.9 billion are activated per token. To better unlock the long context capabilities of the model, MiniMax Text 01 adopts a hybrid architecture that combines Lightning Attention, Softmax Attention and Mixture of Experts (MoE). Leveraging advanced parallel strategies and innovative compute communication overlap methods—such as Linear Attention Sequence Parallelism Plus (LASP+), varlen ring attention, Expert Tensor Parallel (ETP), etc., MiniMax Text 01's training context length is extended to 1 million tokens, and it can handle a context of up to 4 million tokens during the inference. On various academic benchmarks, MiniMax Text 01 also demonstrates the performance of a top tier model. 2. Model Architecture The architecture of MiniMax Text 01 is briefly described as follows: Total Parameters: 456B Activated Parameters per Token: 45.9B Number Layers: 80 Hybrid Attention: a softmax attention is positi…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy