Kimi Linear: An Expressive, Efficient Attention Architecture Paper Code Model (a) On MMLU Pro (4k context length), Kimi Linear achieves 51.0 performance with similar speed as full attention. On RULER (128k context length), it shows Pareto optimal performance (84.3) and 3.98x speedup. (b) Kimi Linear achieves 6.3x faster TPOT compared to MLA, offering significant speedups at long sequence lengths (1M tokens). Overview Kimi Linear is a hybrid linear attention architecture that outperforms traditional full attention methods across various contexts, including short, long, and reinforcement learning (RL) scaling regimes. At its core is Kimi Delta Attention (KDA)—a refined version of Gated DeltaNet that introduces a more efficient gating mechanism to optimize the use of finite state RNN memory. Kimi Linear achieves superior performance and hardware efficiency, especially for long context tasks. It reduces the need for large KV caches by up to 75% and boosts decoding throughput by up to $6\times$ for contexts as long as 1M tokens. We open source the KDA kernel in FLA, and release two versions model checkpoints trained with 5.7T tokens. Model Total Params Activated Params Context Length Do…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy