AxionML Gemma 4 12B FP8 Developed by AxionML for open source serving and deployment use cases. Part of AxionML's effort to provide ready to serve quantized models for the community. This is an FP8 quantized version of google/gemma 4 12B it (11.95B params), using per block 128×128 weight only FP8 (E4M3) with an FP8 KV cache, MSE calibrated. Activations are kept in BF16 — this is deliberate: gemma 4's attention activations carry large per channel outliers, so quantizing activations (per tensor W8A8) measurably degrades the model. Quantizing only the weights is lossless on GSM8K (0.9666 vs 0.9636 BF16) while halving the weight footprint (~24 GB → ~13 GB). Serves on Hopper (H100/H200) and Blackwell. Quantization Details This model was quantized by applying per block (128×128) FP8 (E4M3) to the weights of the linear operators within the transformer blocks. Activations are kept in BF16 (weight only). The KV cache is quantized to FP8 (E4M3) . lm head and the multimodal (vision/audio) embedders are kept in their original BF16 precision. Quantization format FP8 fp8 pb wo — per block 128×128 weight only E4M3, MSE weight calibration Activations BF16 (not quantized) KV cache FP8 (E4M3) Calibra…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy