✨ Gemma4 12B Reasoning Distill (GGUF) ✨ 🐣 Tiny footprint, big brain — local AI for everyone No matter your GPU. No matter your RAM. If you've got ~4.5 GB of VRAM or unified memory free, you can run your own private, offline AI right now. 🚀 Tuned on Opus 4.6, 4.7 & 4.8 reasoning data, it delivers a major leap in reasoning power — whether you're asking questions or writing code. 🧠💻 All local, all yours, no API, no cloud. ⚡ NEW — the MTP version is here! Free speed 🎉 As of June 7, 2026 , mainline llama.cpp just merged Gemma 4 MTP support — so the MTP draft model is now live in the MTP/ folder. Drop it next to any quant and generation gets noticeably faster with identical output (speculative decoding is lossless) — just add a couple of flags. 👉 See ⚡ Speed it up with MTP below. 💚 📦 Pick your size (GGUF quants) Quant Size Vibe 🟢 Q2 K 4.5 GB tiniest — runs almost anywhere 🔵 Q4 K M 6.87 GB the sweet spot 👌 (recommended) 🟣 Q6 K 9.11 GB near lossless ⚪ Q8 0 11.8 GB basically full quality (f16) 22.2 GB full precision (overkill for most) 🧮 "Will it fit?" — context length cheat sheet Rough estimates 🤓 (assumes q8 0 KV cache + ~1.5 GB overhead; use q4 0 KV cache for ≈2× more conte…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy