Mellum2 Thinking — GGUF (MXFP4 MOE) This repository contains a GGUF MXFP4 MOE quantization of JetBrains/Mellum2 12B A2.5B Thinking , ready to run with llama.cpp , Ollama, LM Studio, and other GGUF compatible runtimes. This quantization (MXFP4 MOE): MXFP4 microscaling 4 bit applied to the MoE expert tensors. Smallest footprint, with a modest quality cost (KLD ~0.088, 87% top token agreement). File Size Mellum2 12B A2.5B Thinking MXFP4 MOE.gguf 7.0 GB Mellum 2 Thinking is a Mixture of Experts reasoning model (64 experts, 8 activated per token, 131,072 token context) that emits its chain of thought inside ... blocks before the final answer. For the full model description, evaluation results, and architecture details, see the original model card: JetBrains/Mellum2 12B A2.5B Thinking . Available quantizations Quantization Description Size KLD vs BF16 ↓ Top token match ↑ BF16 16 bit, no quantization (reference) 24.3 GB — — Q8 0 8 bit, effectively lossless 12.9 GB 0.004 97.4% Q6 K 6 bit k quant, very high quality 10.9 GB 0.014 95.1% Q4 K M 4 bit k quant, balanced (recommended) 8.1 GB 0.052 89.8% MXFP4 MOE (this repo) MXFP4 4 bit on MoE experts, smallest 7.0 GB 0.088 87.3% KL divergence an…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy