⚡ Gemma 4 31B IT NVFP4 Turbo GGUF Requires ggml org/llama.cpp 21971 A repackaged nvidia/Gemma 4 31B IT NVFP4 that is 68% smaller in GPU memory and ~2.5× faster than the base model, while retaining nearly identical quality (1 3% loss). Fits on a single RTX 5090 (🎉). Approach Three changes were made: 1. Quantized all self attention weights from BF16 → FP4 (RTN, group size=16, matching modelopt NVFP4 format) 2. Updated architecture to Gemma4ForCausalLM and quantization config accordingly 3. Stripped the vision and audio encoder Everything else is untouched — MLP layers keep NVIDIA's calibrated FP4, embed tokens stays BF16, all norms preserved, so we retain all the nvidia/Gemma 4 31B IT NVFP4 optimizations. Why RTN didn't hurt quality RTN (Round To Nearest) is the simplest quantization method — no calibration data, fully reproducible. It worked here because: FP4 with group size=16 and per group scaling preserves relative weight distributions well Self attention weights tend to be normally distributed near zero, where the FP4 grid has finest resolution (0, 0.5, 1.0, 1.5) MLP layers (more sensitive to quantization) keep NVIDIA's calibrated FP4 embed tokens stays BF16, preventing noise f…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy