gemma 4 12B it AWQ (int4, W4A16) In house AWQ Marlin (4 bit) quantization of google/gemma 4 12B it — the encoder free omni Gemma 4 ( Gemma4UnifiedForConditionalGeneration ): text + reasoning + tool use + vision + audio, global MQA attention, 5:1 sliding:full, ~262K native context. How it was built Packed data free from the QAT base google/gemma 4 12B it qat q4 0 unquantized (Gemma's quantization aware trained release). Because QAT already conditions the weights onto a 4 bit grid, a plain RTN re quantization to AWQ group 128 is near lossless — no calibration corpus needed: Text decoder linears (q/k/v/o proj, mlp gate/up/down proj) → AWQ int4, group size 128. Vision/audio embedders + all norms + tied lm head kept BF16 (modality preservation). Quantization fidelity vs the QAT base: MLP layers cosine 1.0000 , attention projections 0.993–0.996. Quality (2x RTX 3090, SGLang v0.5.12, TP=2) MMLU HumanEval Needle@long ctx 256K tool use Vision Thinking : : : : : : : : : : : : 80% 95% 100% 100% (→95K) ✓ clean Weights are 5.4 GB/rank at int4. Identical text quality to the BF16 base at matched context length. Serving (SGLang) This is the encoder free unified Gemma 4 arch ( gemma4 unified , a tr…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy