Gemma 4 12B it — FP8 dynamic Self quantized FP8 (dynamic) of google/gemma 4 12B it — Google's encoder free omni model (text + image + audio + video). Quantized and benchmarked on an NVIDIA DGX Spark (GB10, sm 121a). TL;DR: 13 GB on disk (from 23 GB BF16), 15.9 tok/s on a GB10 via vLLM, all four modalities intact. Data free — no calibration needed. Benchmark (GB10 / DGX Spark, vLLM 0.22.1 native, single stream decode, warm) Format Disk tok/s (EN/ZH) Omni BF16 23 GB 7.7 yes FP8 dynamic (this) 13 GB 15.9 yes NVFP4 W4A16 7.7 GB 24.9 yes If you want the smallest + fastest build, see the sibling NVFP4 weight only repo. FP8 is the conservative choice (dynamic activations, no calibration, widest kernel support). Accuracy (MMLU + TMMLU+) — near lossless on both languages I scored all three formats on MMLU (English, 57 subjects) and TMMLU+ (Traditional Chinese, 66 subjects) with lm evaluation harness , 5 shot, chat template applied, limit=30 (N ≈ 1,710 EN / 1,980 TC, ±~1.0 pt), through transformers : Format MMLU (EN) TMMLU+ (TC) EN drop TC drop BF16 78.30% 47.21% — — FP8 dynamic (this) 77.95% 46.97% −0.35 −0.24 NVFP4 W4A16 75.56% 41.24% −2.74 −5.97 FP8 is the accuracy preserving choice. Near…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy