GGUF quantization reduces the precision of model weights so a model fits in less memory. The naming looks cryptic — Q4_K_M, Q5_K_S, IQ3_XXS — but it decomposes cleanly, and choosing well is usually the difference between a model that fits your GPU and one that does not. ## Reading the name `Q4_K_M` is three parts. `Q4` is roughly four bits per weight. `K` means a k-quant, which allocates bits unevenly across a block rather than uniformly — more precision where it matters. `S`, `M`, `L` are size variants within that scheme: small, medium, large. `IQ` prefixes are importance-matrix quants. They use a calibration pass to decide which weights tolerate aggressive compression, which buys noticeably better quality at very low bit-widths, at the cost of a slower conversion. ## What the sizes actually look like Concrete numbers from a 20B-parameter model we quantized recently, so the ratios are real rather than illustrative: | Quant | Size | |---|---| | Q2_K | 6.7 GB | | Q3_K_M | 9.1 GB | | Q4_K_M | 12.2 GB | | Q5_K_M | 14.0 GB | | Q6_K | 15.7 GB | | Q8_0 | 20.3 GB | | BF16 | 38.1 GB | The shape matters more than the specific numbers. Going from BF16 to Q8_0 roughly halves the size for a quality difference most workloads cannot detect. Q4_K_M is about a third of BF16 and is the usual production default. Below Q4 the losses become visible, and below Q3 they become obvious. ## How to choose Start from your VRAM, not from a quality preference. Work out how much you actually have free — not how much the card has, since anything else resident takes its share — then pick the largest quant that fits with headroom for the KV cache. If Q4_K_M fits, take it. If only Q3 fits, take Q3 and test on your own inputs before deciding it is acceptable, because degradation is task-dependent: summarization tolerates it far better than code generation or structured output. If nothing sensible fits, the answer is a smaller model at a higher quant rather than a larger model crushed to Q2. A 7B at Q5 will usually serve you better than a 13B at Q2 in the same memory. ## The mistake worth avoiding Do not choose a quant by reading benchmark tables alone. Quantization damage is uneven across tasks, and the aggregate score hides exactly the failure you will care about. Take ten inputs from your real use case, run them through two candidate quants, and read the outputs yourself. It takes twenty minutes and it is the only measurement that reflects your workload. We wrote more about building small evaluation sets like this in the Learn course on LLM foundations.

Understanding GGUF quantization: which quant should you pick?