Gemma 4 12B IT Assistant - GGUF Q8_0
This repository contains a GGUF Q8_0 conversion of google/gemma-4-12B-it-assistant.
This GGUF is intended to be used as an MTP / speculative draft model with a compatible Gemma 4 main model. At the time of upload, this requires the gemma4-mtp llama.cpp branch:
https://github.com/am17an/llama.cpp/tree/gemma4-mtp
Usage
This model is not intended to be run as a standalone chat model. It is intended to be loaded as the draft model with -md.
Required llama.cpp branch:
https://github.com/am17an/llama.cpp/tree/gemma4-mtp
Example llama-server command:
llama-server.exe -m models\gemma-4-12B-it-Q6_K.gguf ^
-md models\gemma-4-12B-it-assistant-Q8_0.gguf ^
--spec-type draft-mtp ^
--spec-draft-n-max 2 ^
--draft-p-min 0.0 ^
-ngl 99 ^
-c 262144 ^
-np 1 ^
-fa on ^
--reasoning off ^
-ctk q8_0 ^
-ctv q8_0 ^
--ctx-checkpoints 128 ^
--temp 0 ^
--top-p 0.95 ^
--top-k 40 ^
--presence-penalty 0.0 ^
--min-p 0.00
Benchmark
Benchmark was run with llama-server.exe from the gemma4-mtp llama.cpp branch.
| Setting | Value |
|---|---|
| Main model | gemma-4-12B-it-Q6_K.gguf |
| Draft model | gemma-4-12B-it-assistant-Q8_0.gguf |
| Required branch | am17an/llama.cpp gemma4-mtp |
| Runs | 3 prompts x 5 measured repeats per mode, with 1 warmup per prompt |
| Generation length | 256 tokens |
| Context | 262144 |
| GPU layers | 99 |
| Flash attention | on |
| KV cache | Q8_0 / Q8_0 |
| Temperature | 0 |
Generation throughput
| Mode | Short gen tok/s | Medium gen tok/s | Long gen tok/s | Mean gen tok/s | Mean speedup vs baseline |
|---|---|---|---|---|---|
| Baseline | 52.08 | 51.94 | 51.10 | 51.71 | - |
| MTP draft n=1 | 57.79 | 56.27 | 58.21 | 57.42 | +11.0% |
| MTP draft n=2 | 57.83 | 59.45 | 61.54 | 59.61 | +15.3% |
| MTP draft n=3 | 52.55 | 57.32 | 65.28 | 58.38 | +12.9% |
In this benchmark, --spec-draft-n-max 2 produced the best average generation throughput across the three prompt types. --spec-draft-n-max 3 was fastest on the long prompt but weaker on the short prompt.
Prompt-processing throughput
| Mode | Short prompt tok/s | Medium prompt tok/s | Long prompt tok/s |
|---|---|---|---|
| Baseline | 1194.85 | 2041.71 | 2840.91 |
| MTP draft n=1 | 1097.50 | 1943.36 | 2545.60 |
| MTP draft n=2 | 1133.98 | 1947.88 | 2542.42 |
| MTP draft n=3 | 1102.01 | 1965.93 | 2572.39 |
Long-prompt prompt-processing values use the first full-ingest/warmup run. Repeated long-prompt runs appeared to reuse cached/prefix state and reported partial prompt-eval timings, so those rows were excluded from the prompt-processing table.
Limitations
These numbers are local throughput measurements from one hardware/software setup. They are not model quality benchmarks and may not generalize to other GPUs, drivers, llama.cpp commits, CUDA versions, context lengths, quantization settings, or sampling settings.
This draft model should be used with a compatible Gemma 4 main model and a llama.cpp build that supports draft-mtp.
Conversion notes
Converted from Hugging Face safetensors to GGUF using the am17an/llama.cpp gemma4-mtp branch.
The upstream config.json used Gemma4UnifiedAssistantForCausalLM / gemma4_unified_assistant. For compatibility with the converter branch, the local config was patched to:
{
"architectures": ["Gemma4AssistantForCausalLM"],
"model_type": "gemma4_assistant"
}
The tokenizer config was also patched by replacing list-valued extra_special_tokens with {} to avoid a local Transformers tokenizer loading error during GGUF export.
Source model
Original model: google/gemma-4-12B-it-assistant