Qwen3.6 27B INT4 AutoRound (Best Recipe) A W4A16 (INT4 weight, FP16 activation) quantization of Qwen/Qwen3.6 27B , produced with Intel's AutoRound. Key difference from other AutoRound quants of this model: This was quantized using the auto round best preset — 1000 iterations and 512 calibration samples instead of the standard 200/128. This preset runs ~4–5× slower but achieves the best possible accuracy at INT4, as it performs a more thorough weight rounding optimization. MTP (speculative decoding) and image/vision inputs work out of the box with no post processing required. TL;DR Base : Qwen3.6 27B (27B dense VLM) Quant : INT4 W4A16, group size 128, symmetric Tool : auto round best (1000 iters, 512 samples, torch.compile) Size : ~18 GB (down from ~54 GB BF16) — 3× reduction MTP : Native Multi Token Prediction head preserved in BF16 — enables native speculative decoding in vLLM (~85–90% draft acceptance, ~2× throughput) Vision : Image inputs work via the MoonViT encoder (weights kept at original BF16/FP16 precision) Quick inference with vLLM (with MTP speculative decoding) Requires vLLM v0.19.1+ with Qwen3 5 MTP support. Set the following environment variables before starting: Remo…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy