gemma-4-A4B-98e-v7-coder-it-GGUF

GGUF quantizations of ManniX-ITA/gemma-4-A4B-98e-v7-coder-it, the loop-fixed code prune of Gemma 4 26B-A4B (128→98 experts/layer, ~20.8B).

All quants made using imatrix with calibration data v5. The imatrix.dat used is included in this repo for reproducibility/audit; mmproj-gemma4.gguf is the shared Gemma 4 SigLIP vision tower (untouched by pruning) for multimodal use.

Quantizations — score, size & bits-per-weight

Each published tier was scored on HumanEval+ (164) and MultiPL-E-100 (llama.cpp, deployment sampler — min_p), as a quant-vs-quant comparison of how each tier holds the model's code ability. The table shows, per tier, the code score, the exact size, and the true bits-per-weight (bpw = 8 × bytes ÷ 19,877,953,946). ⭐ marks a recommended pick.

Recommended picks:

• Q4_K_M ⭐ (13.24 GB) — recommended default — ties the best HE+ in the sweep (92.68%) at 13.2 GB.
• Q4_K_S ⭐ (12.21 GB) — same 92.68% HE+ at 12.2 GB if you want the smaller 4-bit K-quant.
• CD-Q2_K ⭐ (8.82 GB) — smallest tier still in the ~90% band — 89.63% HE+ / 86.0% MPE at 8.8 GB.

Reads: HE+ holds in the 89–93% band and MPE in the 86–91% band across the whole K-quant / CD ladder — the 4-bit K-quants (Q4_K_M / Q4_K_S at 92.68%) actually top the sweep, and even the 3.55-bpw CD-Q2_K stays at ~90%. The CD-Q2_K ContribDynamic per-layer body is the recommended low-bit path; the pruned MoE degenerates on a plain 2-bit or an IQ-family body at this size (token-salad), so those tiers are not offered.

Head-to-head by file size — v7-coder vs Qwen2.5-Coder-14B (iso-disk)

Pairing by tier name is misleading — v7-coder is a ~20.8B-total MoE and Qwen2.5-Coder-14B is a 14.7B dense model, so the same tier name lands at a different file size. The fair comparison is iso-disk: at a given GB budget, which model scores higher on HumanEval+? Qwen GGUFs are bartowski's Qwen2.5-Coder-14B-Instruct-GGUF; its ladder sits at 83–85% across the whole stack. v7-coder HE+ is the deployment-sampler per-tier sweep above; Qwen HE+ is the same-stack reference.

Reads:

1. Iso-disk ~10.5 GB. v7-coder Q3_K_M (10.51 GB / 4.23 bpw / 91.46%) vs Qwen Q5_K_M (10.51 GB / 5.72 bpw / 83.54%): +7.92pp at the exact same file size, −1.49 bpw.
2. Sub-9 GB code-grade. CD-Q2_K (8.82 GB / 3.55 bpw / 89.63%) holds the ~90% HE+ band ~0.2 GB smaller and ~1.3 bpw lower than Qwen's best (85.37%, Q4_K_M, 8.99 GB).
3. Every band wins at lower bpw. Across the ladder the MoE uses 1.5–4 bpw less than the dense Qwen tier at the same disk and still scores higher on HumanEval+ — the point of the iso-disk framing.

CD recipes are open-source — generator at omnimergekit/scripts/generate_cd_maps.py.

How to Use

With llama.cpp:

llama-server -m gemma-4-A4B-98e-v7-coder-it-Q4_K_M.gguf -c 32768 -ngl 99 \
    --jinja \
    --reasoning-format deepseek --reasoning-budget 8192 \
    --temp 1.0 --top-k 64 --top-p 0.95 --min-p 0.05 \
    --repeat-penalty 1.02 --repeat-last-n 2048

The --reasoning-budget flag is required for Gemma 4 thinking — without it the model emits malformed channel tokens. Keep -c several times larger than the reasoning budget — with -c equal to the budget the thinking phase can fill the whole window and the answer degenerates.

Recommended sampling — --repeat-penalty 1.02 --repeat-last-n 2048 is the field-tested sweet spot for long agentic / tool-calling sessions (validated with opencode driving multi-turn coding work): it suppresses the intermittent repetition loops without side effects. Stronger penalties (1.05–1.1) also stop the loops but cause premature end-of-turn — the model announces a step and then stops mid-task. Narrower windows (64–1024) at 1.02 let long-period loops through; keep the full 2048. The remaining sampler values above are the Gemma 4 vendor defaults baked into the GGUF, stated explicitly so CLI defaults don't override them.

With ollama: ollama pull mannix/gemma4-98e-v7-coder:Q4_K_M (:latest = Q4_K_M; :vision-<tier> pairs the tier with the SigLIP vision tower).

Chat template & end-of-turn tokens (metadata refresh, 2026-07-30)

Every tier in this repo has been re-uploaded with corrected metadata — the rollout completed 2026-07-30 and covers all 14 tiers: Q8_0, Q6_K_L, Q6_K, Q5_K_L, Q5_K_M, Q4_K_L, Q4_K_M, Q4_K_S, IQ4_NL, IQ4_XS, Q3_K_L, Q3_K_M, CD-Q2_K, F16. Every tier was verified after patching (template 19,177 B / md5 8119c2dc…, eos=106, eot=1, bos=2, tool-call macros present, tensor count unchanged) and the published file's sha256 re-checked against the patched file. The matching mannix/gemma4-98e-v7-coder ollama tags were re-pushed as well — all 13 text tiers and their 13 vision-<tier> counterparts (F16 is HF-only, by design).

The tensor payload is byte-identical — only the GGUF KV header changed — so sizes and scores are unaffected, but you need to re-download to pick the fix up. A local file still reporting eos_token_id = 1 (gguf_dump / llama-server startup log) is a stale copy from before the refresh.

1. Chat template — agentic loop fix, rebased onto Google's current template. The embedded tokenizer.chat_template is now our loop fix applied on top of Google's current Gemma 4 template (upstream revision 2026-07-20, 18,683 B). The result is 19,177 B, md5 8119c2dcd5e62a4a6b79301ab13ac81d, and it is also published at the repo root as chat_template.fixed.jinja for anyone serving the bf16 weights.

The bug it fixes: the stock template re-injects earlier assistant turns' thinking content back into the prompt on every turn. In long agentic / tool-calling sessions that feeds the model its own reasoning back to itself and drives the repetition loops. Google's current 18,683 B template is still affected — its thinking gate carries an unconditional "index past the last user message" disjunct — so moving to a fresh upstream template does not remove the need for this fix. The rebase leaves Google's newer preserve_thinking flag intact (default false).

llama.cpp only uses the embedded template when you pass --jinja (added to the command above). Without it llama.cpp falls back to its own built-in Gemma 4 formatter and the fix does not apply. The recommended sampler above is still recommended — the template fix and --repeat-penalty 1.02 address different halves of the loop problem.

2. End-of-turn tokens. eos_token_id is now 106 (<turn|>) and eot_token_id 1 (<eos>), which gives llama.cpp the end-of-generation set {106, 1, 50} — matching generation_config.json ([1, 106, 50]) in the source repo. Previously every tier shipped eos_token_id = 1 with no eot, so stopping depended on stop strings; the model can now end its turn on its own token.

ollama caveat. ollama ≥ 0.32 formats Gemma 4 with a compiled-in gemma4 renderer and parser and never reads the GGUF's jinja template, so the mannix/gemma4-98e-v7-coder tags receive the EOG fix only. That is expected, not a defect: ollama show --modelfile printing TEMPLATE {{ .Prompt }} next to RENDERER gemma4 / PARSER gemma4 is the correct state. For the template fix, serve the GGUF with llama.cpp --jinja.

Gemma 4 A4B 98-Expert v7-coder — loop-fixed code prune (~20.8B)

Eval complete (Q6_K / llama.cpp, greedy, same host). Every cell in the scoreboard is read from summary.json under the cohort-pinned greedy recipe (temperature 0.0, top_p 1.0, top_k 0). The 128e, v6-coder and v7-coderx columns are the matching same-host Q6_K runs. The GGUF and NVFP4A16 formats are deployment targets and are not separately benchmarked (cohort policy) — the Q6_K column is representative.

Headline — the cohort's balanced code build. v7-coder leads the cohort on the easier-but-broad LiveCodeBench-medium slices (LCB-55-v4 98.18%, LCB-100-v4 94.0%) and on HumanEval (98.17%), ties the cohort top on MATH-500 (95.0%), and takes AIME (80.0%). On the all-hard LiveCodeBench-77 set — the most discriminating LCB slice — it scores 84.42% (128e 79.22%, v7-coderx 85.71%), just behind the code-maximal sibling. This is the loop-fixed build: it force-keeps the agentic loop-protection experts and replaces the earlier looping fs2440 prune. Like its sibling it spends the prune budget on graduate science — GPQA-diamond sits at 51.52% (no targeted_gpqa term; ≈ v7-coderx 51.01%). If you need the hardest-code lean (all-hard LCB + HE+), see the sibling v7-coderx.

A research checkpoint that prunes the unpruned Gemma 4 26B-A4B-it (128 experts/layer, top-8 + shared, 30 layers) down to 98 experts per layer. The fkbroad drop map (generate_drop_map_v5) up-weights generic-code (3×) and LiveCodeBench-medium (2×) with no science or multilingual targeting, and force-keeps the agentic loop-protection experts (agentic_eog, 46 experts, 0/46 dropped) so the served model does not loop. Same 98e shape, same router, same attention, same norms as the rest of the cohort, plus the mandatory shared-FFN α=1.2 upweight all coder variants carry. No per-layer floor clamp and no DERN fold.

Benchmarks

Q6_K · llama.cpp · greedy (temperature 0.0, top_p 1.0, top_k 0), all four models scored on the same host from summary.json. Row-max in bold. This repo = v7-coder.

_{Metrics: GPQA & GSM8K = exact_match flexible-extract · MATH500 = math_verify · ARC & AIME = exact_match · IFEval = prompt_level_strict_acc · HumanEval/+ = pass@1 chat-extract · LCB-55/100 & MultiPL-E = pass@1. 128e uses the lcb_medium_55/100 templates; the prunes use lcb_medium_*_v4 (corrected harness, equivalent task).}

v7-coder is the balanced code sibling: it tops the cohort on LCB-medium and HumanEval and ties on MATH/AIME, while v7-coderx leads the all-hard LCB-77 and HE+. Both pay the budget on graduate science (GPQA ≈ 51) and the easier ARC axis.

LiveCodeBench across problem sets

The code score depends on the LiveCodeBench slice. All cells are the same greedy Q6_K / imat-Q6 llama.cpp stack (build provenance verified per run); v4-55/100 mirror the 9-bench above. The all-hard 77q set is the most demanding and the most discriminating across the cohort.

Coder-field comparison — v7-coder vs Qwen2.5-Coder-14B / 7B + Qwen3.5-9B (Q6_K, llama.cpp, greedy)

The 9 canonical benches + MultiPL-E-100, all on the identical llama.cpp Q6_K / greedy recipe (reasoning models served with --reasoning-format deepseek --reasoning-budget 12288 --parallel 2). Architectures differ — this is a same-harness comparison, not a same-class one:

• v7-coder — Gemma-4 26B-A4B MoE pruned to 98 experts (~20.8B total, ~A4B active), reasoning.
• Qwen2.5-Coder-14B / 7B-Instruct — dense, non-reasoning code specialists (bartowski Q6_K).
• Qwen3.5-9B — dense reasoning model (bartowski Q6_K).

† Qwen2.5-Coder-14B HumanEval+ / LCB-medium-55 are the same-stack GGUF HE+ sweep numbers (not re-run in this chain). All Qwen cells are the same-host reference runs used on the v6-coder card — Qwen is a fixed reference, so the columns are identical across the cohort; only the Gemma column changes.

Note on Qwen3.5-9B. Qwen3.5-9B is a verbose, slow thinking model: it emits long <think> reasoning chains (often ≥1900 tokens even on a trivial GSM8K question), so it runs several× slower per question than the non-reasoning Qwen2.5-Coder models — well beyond what its 9B size would suggest. Its GSM8K / MATH-500 / GPQA cells were re-run after a harness fix (under batched, reasoning-parsed serving the verbose thinking intermittently left the final answer inside the reasoning block, mis-scored as empty content).

Recipe (summary)

98e prune from 128e via the fkbroad code recipe (generate_drop_map_v5: generic_code 3×, targeted_lcb_medium_55 2×, all other targeting 0; target=98, protect_top=16, alpha=2.0, strategy=max + breadth_bonus, no per-layer floor clamp), then the agentic loop-protection force-keep (agentic_eog, 46 experts, 0/46 dropped — the loop fix that replaces fs2440), then the mandatory shared-FFN α=1.2 upweight. No targeted_gpqa term and no DERN fold. v7-coder uses code/LCB weight 3×/2×; the sibling v7-coderx uses 4×/3×. Full recipe is on the bf16 card.

Intended use & limitations

Compact (~13 GB at Q4_K_M, single 12–16 GB GPU) Gemma 4 checkpoint for agentic coding and code reasoning. Serve with the reasoning parser enabled. A research prune, not an official Google release; generic_multilingual is de-weighted (0×) and graduate science (GPQA) is a budget axis (51.52% vs 128e 67.17%). Prefer Q4_K_M or higher.

Lineage

128e → (v4 → v5 → v6-coder code line) → v7 competence-map rebuild → fkbroad code3/lcb2 selection + agentic loop-protection force-keep = v7-coder (loop-fixed; supersedes fs2440). Built and evaluated on the omnimergekit toolchain.