Qwen3.6 27B INT4 AutoRound A W4A16 (INT4 weight, FP16 activation) quantization of Qwen/Qwen3.6 27B , produced with Intel's AutoRound. TL;DR Base : Qwen3.6 27B (27B dense VLM, Apr 21 2026) Quant : INT4 W4A16, group size 128, symmetric Tool : auto round (default recipe, 200 iters, torch.compile) Size : 18 GB (down from ~54 GB BF16) — 3x reduction MTP preserved : The native Multi Token Prediction head is kept in BF16, enabling native speculative decoding in vLLM (≈90% draft acceptance in our tests, ~2x throughput) Accuracy : Default AutoRound recipe preserves quality well; layer norm weights, router layers, RMSNorm, linear attn.in proj a/b , and MTP's fusion fc are kept unquantized (they're small and benefit from full precision) Quick inference with vLLM (with MTP speculative decoding) Requires vLLM that supports Qwen3 5 MTP (most recent nightlies — tested with eugr/spark vllm docker fork 0.19.1rc1.dev39+g7055d32a7 ): Notes: kv cache dtype tq t4nc (TurboQuant 4 bit) halves KV memory vs fp8. Use kv cache dtype fp8 for mainline vLLM without the TurboQuant fork. compilation config.cudagraph mode none is currently needed on Blackwell consumer (SM120/SM121) GPUs — CUDA graph capture hits a…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy