🧊 Gemma 4 31B it · AWQ · 2 bit · GGUF AWQ + imatrix (hybrid) 📦 8.88 / 10.17 / 10.22 GiB IQ2 XS / IQ2 M / Q2 K S + QAT sourced IQ2 XS / Q2 K S 🏗️ llama.cpp f3e1828 ⚡ + MTP drafter (speculative decode) 🏅 medKLD 1.09 · top p 49.0% · PPL 120 🧊 What this is Three aggressively compressed (under 3 bits per weight) quantizations of google/gemma 4 31B it . Before quantizing, each linear layer is rescaled by a per channel factor (the AWQ trick: Activation aware Weight Quantization) so that outlier channels don't blow up the 2 bit codebook. We search for the best rescale strength on a calibration text, but only keep an aggressive per tensor choice if it also improves loss on a separate held out text it never saw during the search; otherwise we fall back to a safer default. The rescale is absorbed into the preceding RMSNorm layer, so the file is a plain GGUF with no custom runtime and no extra inference cost . 📉 ~5 6.5x smaller size (Gb) At ~2 bits per weight, these quants are under 11 GiB on disk vs 57.2 GiB for FP16. 🎯 up to ~50% top p at 2 bit and ~80% at 4 bit Top token agreement with FP16: 51.1% on vanilla Q2 K S AWQ, 49.4% on QAT Q2 K S AWQ, 48.9% on QAT IQ2 XS AWQ — best in class…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy