GDN primed HQwen3 8B Instruct GDN primed HQwen3 8B Instruct is a Hybrid language model consisting of 50% Attention layers and 50% Gated DeltaNet (GDN) layers, primed from Qwen3 8B using the Hybrid Model Factory Priming pipeline. The model is instruction tuned and supports context lengths up to 128K tokens. GDN is a State Space Model layer with constant memory and linear compute cost in the sequence length. By combining Attention with GDN, our Hybrid model achieves up to 2× faster inference at long contexts while closely matching the base Transformer's quality . Links 📄 Gated DeltaNet paper (ICLR 2025) 💻 GitHub: Hybrid Model Factory Why Hybrid? Each Primed Hybrid model is initialized from a base Transformer by converting a portion of its Attention layers into State Space Model (SSM) layers that maintain a fixed size recurrent state instead of a growing KV cache. At a 50% Hybrid ratio, roughly half the KV cache (which grows linearly with sequence length) is replaced with fixed size SSM state. The practical benefits: Higher throughput at long contexts — less memory on KV cache means more memory for batching More concurrent sequences — ~2× as many concurrent sequences before hitting…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy