gemma-4-12B-it AWQ (int4, W4A16)
In-house AWQ-Marlin (4-bit) quantization of google/gemma-4-12B-it — the encoder-free omni Gemma 4 (Gemma4UnifiedForConditionalGeneration): text + reasoning + tool-use + vision + audio, global-MQA attention, 5:1 sliding:full, ~262K native context.
How it was built
Packed data-free from the QAT base google/gemma-4-12B-it-qat-q4_0-unquantized (Gemma's quantization-aware-trained release). Because QAT already conditions the weights onto a 4-bit grid, a plain RTN re-quantization to AWQ group-128 is near-lossless — no calibration corpus needed:
- Text-decoder linears (q/k/v/o_proj, mlp gate/up/down_proj) → AWQ int4, group_size 128.
- Vision/audio embedders + all norms + tied
lm_headkept BF16 (modality preservation). - Quantization fidelity vs the QAT base: MLP layers cosine 1.0000, attention projections 0.993–0.996.
Quality (2x RTX 3090, SGLang v0.5.12, TP=2)
| MMLU | HumanEval | Needle@long-ctx | 256K tool-use | Vision | Thinking |
|---|---|---|---|---|---|
| 80% | 95% | 100% | 100% (→95K) | ✓ | clean |
Weights are 5.4 GB/rank at int4. Identical text quality to the BF16 base at matched context length.
Serving (SGLang)
This is the encoder-free unified Gemma 4 arch (gemma4_unified, a transformers-5.10.dev architecture). Serving on SGLang needs the back-port patches 042–048 from 2x-3090-GA102-300-A1-sglang-inference (loader + vendored config/processor + the hybrid-SWA TP>1 KV-pool routing fix + the processor __call__ image-token expansion). On Ampere (sm_86), use --attention-backend triton (FlashInfer rejects the 512-dim global head) and --disable-cuda-graph.
License
Governed by the Gemma Terms of Use. Derived from google/gemma-4-12B-it.