Important Update: Unsloth assistant GGUFs Unsloth has released working Gemma 4 12B instruct assistant models that work with upstream llama.cpp for mtp. Check them out here: https://huggingface.co/unsloth/gemma 4 12b it GGUF/tree/main/MTP Generally you can find them at Unsloth's model pages in the files section where there will be an MTP folder containing the assistant model GGUFs. Originally this model repo had experimental GGUF quantized versions of the gemma 4 12B it assistant from google, but I have since then updated them to be quantizations of Unsloth's MTP drafters which are compatible with the latest llama.cpp out of the box. Gemma 4 12B IT QUANTIZED MTP GGUFs This model repo contains various GGUF format quantizations of Unsloth's gemma 4 12b it BF16 MTP.gguf. I did not include BF16, F16, or Q8 0 because those are available from Unsloth at that link. All of these quantizations were made using llama quantize from llama.cpp. These models are just the MTP heads which are meant to be used alongside the base model, not on their own. Example Usage
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy