BMOJOF primed HQwen3 8B Instruct BMOJOF primed HQwen3 8B Instruct is a Hybrid language model consisting of 50% Attention layers and 50% B'MOJO F layers, primed from Qwen3 8B using the Hybrid Model Factory Priming pipeline. The model is instruction tuned and supports context lengths up to 128K tokens. The B'MOJO F layer is a hybrid layer that couples sliding window Attention (eidetic memory for precise local context) with a State Space Model (fading memory for a compressed summary of the unbounded past) within a single layer, rather than alternating separate Attention and SSM layers across the model. This makes it unique among hybrid architectures because each B'MOJO F layer simultaneously maintains both forms of memory, exact recall of recent tokens and an approximate, ever updating representation of the full history, allowing the model to flexibly trade off between high fidelity local retrieval and efficient long range context compression. By combining Attention with B'MOJO F, our Hybrid model achieves up to 1.75× faster inference at long contexts while closely matching the base Transformer's quality . Links 📄 B'MOJO paper (NeurIPS 2024) 💻 GitHub: Hybrid Model Factory Why Hybrid…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy