Qwen3.6 27b gptq int4 GPTQ INT4 quantization of Qwen/Qwen3.6 27B . 3× smaller. ~2.4× faster. Author: Prashant Takale Model Compression BF16 baseline GPTQ INT4 (this model) VRAM at load ~54 GB ~14 GB (3.9× smaller) Bits / weight 16 4.29 (3.7× fewer) Benchmarks Note: MMLU Redux uses a 1500 sample subset; other tasks are full. Decoding/prompts/filters are lm eval harness defaults, so absolute scores may differ from the official Qwen3.6 27B numbers. The goal is the BF16↔INT4 delta under identical conditions, not exact replication of the baseline. Both models evaluated under identical conditions with lm evaluation harness: greedy decoding ( temperature=0 ), enable thinking=False , seed=0. Long CoT tasks use max gen toks=4096 ; HumanEval served via /v1/completions (raw, no chat template) so the harness's \\ndef / \\nclass stop sequences fire correctly. Inference Performance Single stream measurement on the same hardware, identical request (337 input / 42 output tokens): INT4 delivers ~2.4× higher throughput and ~2.4× lower latency at single stream — the bandwidth savings from 4 bit weights translate almost 1:1 into decode time speed up (output tok/s and TPOT). Quantization recipe Setting…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy