GKA primed HQwen3 8B Instruct GKA primed HQwen3 8B Instruct is a Hybrid language model consisting of 50% Attention layers and 50% Gated KalmaNet (GKA) layers, primed from Qwen3 8B using the Hybrid Model Factory Priming pipeline. The model is instruction tuned and supports context lengths up to 128K tokens. GKA (pronounced as gee ka) is a State Space Model layer inspired by the Kalman Filter that solves an online ridge regression problem at test time, with constant memory and linear compute cost in the sequence length. By combining Attention with GKA, our Hybrid model achieves up to 2× faster inference at long contexts while closely matching the base Transformer's quality . Links 📄 Gated KalmaNet paper (CVPR 2026) 💻 GitHub: Hybrid Model Factory Why Hybrid? Each Primed Hybrid model is initialized from a base Transformer by converting a portion of its Attention layers into State Space Model (SSM) layers that maintain a fixed size recurrent state instead of a growing KV cache. At a 50% Hybrid ratio, roughly half the KV cache (which grows linearly with sequence length) is replaced with fixed size SSM state. The practical benefits: Higher throughput at long contexts — less memory on KV…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy