Qwen3.6 27B INT8 AutoRound This is an unofficial INT8 quantized version of the Qwen3.6 27B. It was created using AutoRound. Available versions There are two versions. Main branch one is a little bit smaller by quantizing self attn, and disabling group size at the cost of the model's intelligence. For users with 48GB VRAM, just using Main branch is recommended. If you have more than that, gs128 branch might be better. The performance difference in practical use is minimal. Quantization details Field Main branch gs128 branch Base Qwen/Qwen3.6 27B Qwen/Qwen3.6 27B Method AutoRound ( intel/auto round ), custom recipe AutoRound ( intel/auto round ), default recipe Scheme W8A16 W8A16 Bits 8 8 Group size 1 128 Symmetric yes yes Unquantized layers visual , mtp , linear attn , embed tokens , lm head visual , mtp , self attn , linear attn , embed tokens , lm head Calibration samples 128 128 Iterations 1000 200 Batch size 8 8 torch.compile enabled enabled Size 36.8GB 38.8GB GPU used for quant 2× RTX 3090 2× RTX 3090 For more information, please check quantize.py. Evaluation Results (KLD) Lower values indicate less degradation caused by quantization. Main branch is used for the evaluation. KLD…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy