[!IMPORTANT] Multi Token Prediction (MTP) Layers Grafted on INT8 Quantization This repo is the TheHouseOfTheDude/Qwen3.6 27B INT8 W8A16 PTQ quantization with unquantized BF16 MTP layers grafted on top . Base quantization: TheHouseOfTheDude Qwen3.6 27B INT8 — INT8 weights, FP16/BF16 activations via compressed tensors (pack quantized format). See the original repo for quantization details and KLD results. MTP layers: Taken from Qwen/Qwen3.6 27B in full BF16 precision and added as a second safetensors file ( model 00002 of 00002.safetensors ). Chat template (v9): Updated from froggeric/Qwen Fixed Chat Templates. Fixes broken tags, misplaced tool calls inside think blocks, and other structural issues that break agentic coding and OpenClaw usage. Why BF16 for MTP? MTP layers were excluded from the original quantization recipe ( ignore: ['re:. mtp. '] ). Keeping them unquantized avoids any loss of draft token quality while adding only ~850 MB of overhead. config.json includes re:. mtp. in the quantization config.ignore list so vLLM skips quantization wrapping for MTP Linear layers during loading. To serve with MTP enabled, use: Qwen3.6 27B INT8 MTP Post Training Quantized (PTQ) version o…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy