Mellum2 Thinking — GGUF (Q4 K M) This repository contains a GGUF Q4 K M quantization of JetBrains/Mellum2 12B A2.5B Thinking , ready to run with llama.cpp , Ollama, LM Studio, and other GGUF compatible runtimes. This quantization (Q4 K M): 4 bit k quant (medium). Strong quality/size trade off (KLD ~0.052, 90% top token agreement) — a good default. File Size Mellum2 12B A2.5B Thinking Q4 K M.gguf 8.1 GB Mellum 2 Thinking is a Mixture of Experts reasoning model (64 experts, 8 activated per token, 131,072 token context) that emits its chain of thought inside ... blocks before the final answer. For the full model description, evaluation results, and architecture details, see the original model card: JetBrains/Mellum2 12B A2.5B Thinking . Available quantizations Quantization Description Size KLD vs BF16 ↓ Top token match ↑ BF16 16 bit, no quantization (reference) 24.3 GB — — Q8 0 8 bit, effectively lossless 12.9 GB 0.004 97.4% Q6 K 6 bit k quant, very high quality 10.9 GB 0.014 95.1% Q4 K M (this repo) 4 bit k quant, balanced (recommended) 8.1 GB 0.052 89.8% MXFP4 MOE MXFP4 4 bit on MoE experts, smallest 7.0 GB 0.088 87.3% KL divergence and top token agreement are measured against the B…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy