🧰 1. Files & comparison

Corpora under calibration_data/:

All rows benched on the same corpus.eval.txt, same llama.cpp build, ctx=4096. AWQ calibrate uses proxy_tokens=1024, ctx=4096; the Q2_K_S AWQ rows use the q2k_b16 codebook proxy and the IQ2_M AWQ row uses q2k_b16 base + iq3_s mix for the IQ3_S-bumped tensors. KLD and top_p are measured against google/gemma-4-31B-it FP16 (so QAT and vanilla rows are directly comparable). KLD column is median for robustness to per-token tails.

Original google/gemma-4-31B-it source

Here's the transposed (orientation-flipped) version of your table — columns are now quantization variants and rows are the metrics:

QAT google/gemma-4-31B-it-qat-q4_0-unquantized source

Here's the transposed version of the QAT table:

Same approach as before — metrics run down the left side, each variant gets its own column, bold formatting and all links preserved with shortened link text for readability.

No IQ2_M-from-QAT row. IQ2_M quantization of the QAT weights collapses to garbage across every arm tested (PPL ≈ 2×10¹⁰, top_p = 0%) — the IQ2_M codebook and QAT-shaped weights are geometrically incompatible. Use the vanilla-source IQ2_M_AWQ, or step to the QAT-sourced Q2_K_S / IQ2_XS.

📊 2. Metrics deep-dive

We benchmark each quant using a subset of the MMLU-pro dataset (50 questions from each category) and using 50 tool calling samples from real-usage logs extracted from claude code, qwen code and opencode. Each metric is reported as the average of five reps and in the case of MMLU-pro we use 2-shot prompting to speed up the evaluation (in a separate test we found the accuracy trends are very similar with >2 shots, but the eval time balloons). The performance factor is measured as the average ratio between different categories relative to the QAT Gemma-4-31b-Q4_0 model, recently released by Google. In most cases, we find that with proper calibration, the 2-bit quantizations can achieve performance within ~90% of the baseline in both tool calling and general knowledge. If a model is quantized at training it tends to have better performance once compressed further (e.g. to 2-bit) but with proper calibration (e.g. imatrix, awq, gptq, etc.), the gap can be narrowed significantly. The line between PPL, KLD and Top(p) is a bit blurry in relation to actual performance.

🔬 3. How they were made

y  =  W · a  =  (W · diag(s))  ·  (diag(1/s) · a)
              └──────┬──────┘    └──────┬──────┘
              quantize this       fold into prev RMSNorm gain

Toolchain: AWQ + imatrix orchestrated by quant-tuner; final quantization with llama-quantize from llama.cpp pinned to commit f3e1828.

🚀 4. Usage

Ollama

Pull and run directly from Hugging Face — no manual download needed:

# Best overall (QAT source, lowest KLD)
ollama run hf.co/pearsonkyle/gemma-4-31B-it-awq-2bit-GGUF:QAT_Q2_K_S_AWQ

# Best vanilla (highest same_top_p)
ollama run hf.co/pearsonkyle/gemma-4-31B-it-awq-2bit-GGUF:Q2_K_S_AWQ

# Best compact (8.88 GiB, QAT + AWQ)
ollama run hf.co/pearsonkyle/gemma-4-31B-it-awq-2bit-GGUF:QAT_IQ2_XS_AWQ

# 4-bit reference (near-FP16 quality, 15.6 GiB)
ollama run hf.co/pearsonkyle/gemma-4-31B-it-awq-2bit-GGUF:IQ4_XS

All available tags:

Building llama.cpp from source (GPU)

To run these models on the GPU, build llama.cpp with CUDA support:

# Install dependencies and clone repo
apt-get update && apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp

# Build with CUDA (set -DGGML_CUDA=OFF for CPU/Metal)
cmake llama.cpp -B llama.cpp/build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-server

# Move binaries
cp llama.cpp/build/bin/llama-* llama.cpp/

Running the server

# Start the server with your chosen model
 ./llama-server \
    --model gemma-4-31B-it-IQ2_M.gguf \
    --ctx-size 16384 \
    --n-gpu-layers 999 \
    --split-mode layer \
    --flash-attn on \
    --cache-type-k q8_0 \
    --cache-type-v q8_0 \
    --parallel 1 \
    --batch-size 2048 \
    --ubatch-size 512 \
    --host 0.0.0.0 \
    --port 1234

Querying via the OpenAI compatible API

import json, base64, urllib.request

def ask(content, max_tokens=256):
    body = {
        "messages": [{"role": "user", "content": content}],
        "max_tokens": max_tokens,
        # Gemma 4 is a thinking model. Set this to False (or raise max_tokens),
        # otherwise the reply lands in reasoning_content and "content" is empty.
        "chat_template_kwargs": {"enable_thinking": False},
    }
    req = urllib.request.Request("http://127.0.0.1:1234/v1/chat/completions",
                                 json.dumps(body).encode(),
                                 {"Content-Type": "application/json"})
    return json.loads(urllib.request.urlopen(req).read())["choices"][0]["message"]["content"]

b64 = lambda p: base64.b64encode(open(p, "rb").read()).decode()

# Text
print(ask("What is 1+1?"))

⚡ Speculative decoding with the MTP drafter

This package ships one Multi-Token-Prediction (MTP) drafter — a small speculative draft model that predicts several future tokens from the target's last hidden state. It runs as a separate --model-draft (arch gemma4-assistant), so none of the AWQ trunk files are modified.

One drafter serves every quant here (IQ2_XS / IQ2_M / Q2_K_S, vanilla and QAT). It keys off the base model's hidden size and vocab — not the quantization — so there is nothing per-quant to download or attach. It also cannot be merged into a single file: a GGUF carries exactly one general.architecture, and the drafter (gemma4-assistant) is a distinct architecture from the trunk (gemma4), which has no in-graph MTP layers. (This is unlike Qwen3.5, where the MTP bundles into the trunk's own arch.) Keep the drafter beside whichever trunk you run.

./llama-server \
    --model gemma-4-31B-it-Q2_K_S_AWQ.gguf \
    --model-draft mtp-gemma-4-31B-it.gguf \
    --spec-type draft-mtp --spec-draft-n-max 2 \
    --ctx-size 4096 --n-gpu-layers 999 --flash-attn on \
    --host 0.0.0.0 --port 1234

Tune --spec-draft-n-max to your hardware. Speculative decoding only helps when the target decode is latency-bound; on a bandwidth-bound 2-bit trunk the draft overhead can outweigh the gain at large draft depths. Measured on an Apple M4 Max (Metal), vanilla Q2_K_S_AWQ trunk:

On Metal, n-max 1–2 gives ~+7%; n-max ≥ 4 regresses. On a high-end discrete GPU (e.g. B200) the same drafter yields ~1.4× (per Unsloth's measurements), so push n-max higher there. Acceptance stays high (0.7–0.9) even on these 2-bit trunks.

🪪 5. License & attribution

• Inherits the Gemma Terms of Use from the base model.
• Base weights — vanilla-source files: google/gemma-4-31B-it.
• Base weights — QAT-source files (qat-*): google/gemma-4-31B-it-qat-q4_0-unquantized, Google's quantization-aware-trained FP16 checkpoint. The Q4_0 reference row in §1 links to the matching official GGUF release.
• Calibration + AWQ scaling + quantization performed locally with Quant-Tuner; vendored llama.cpp at commit 32782998.
• Calibration data (usage logs) scraped using LogMiner.
• MTP drafter (mtp-gemma-4-31B-it.gguf, MTP/*): derived from Google's google/gemma-4-31B-it-assistant head; the mainline-loadable gemma4-assistant GGUF was published by unsloth/gemma-4-31B-it-GGUF and is redistributed here under the same Gemma terms. Run requires llama.cpp ≥ 2026-06-07 (PR #23398).