Qwen3.6-27B-Quark-W8A8-INT8
W8A8 INT8 quantized version of Qwen/Qwen3.6-27B using AMD Quark.
Model Details
| Base Model | Qwen/Qwen3.6-27B |
| Architecture | Qwen3_5ForConditionalGeneration (hybrid attention + ViT) |
| Parameters | 27B language tower (quantized) + 27-layer ViT (BF16, unquantized) |
| Layers | 64 hybrid (16 full_attention + 48 linear_attention GatedDeltaNet) + 1 MTP head |
| Quantization | W8A8 INT8 (per-channel weight + per-token dynamic activation) |
| Quantizer | AMD Quark 0.11.1 (pack_method='reorder', vLLM-native key naming) |
| Model Size | ~29 GB (single safetensors) |
| Original Size | ~52 GB (BF16) |
| Compression | ~1.8x size reduction |
Quantization Scheme
| Component | dtype | Granularity | Mode |
|---|---|---|---|
| Linear weight (text decoder) | INT8 | per-channel (ch_axis=0) | symmetric, static |
| Linear activation | INT8 | per-token (ch_axis=1) | symmetric, dynamic |
lm_head | BF16 | - | unquantized |
embed_tokens | BF16 | - | unquantized |
| Vision tower (27 ViT blocks) | BF16 | - | unquantized |
MTP head (mtp*) | BF16 | - | unquantized |
Accuracy
GSM8K full 1319-question test split (vLLM, temperature=0, concurrency=16,
max_tokens=1024, chat_template_kwargs.enable_thinking=false):
| Model | Accuracy | Correct |
|---|---|---|
Qwen/Qwen3.6-27B (BF16 baseline) | 96.74% | 1276 / 1319 |
| This model (Quark W8A8 INT8) | 96.74% | 1276 / 1319 |
Net accuracy delta vs BF16: 0.00 pp.
Although the totals match exactly, the two models diverge on individual questions: only 38 / 1319 generations are token-identical, and the correct-set Jaccard is 0.9891 (1269 common correct, BF16 wins 7 unique, INT8 wins 7 unique — they cancel out). This is the typical W8A8 INT8 pattern: small per-token numerical drift causes reasoning paths to fork, but the accuracy averages out with no systematic degradation.
Eval setup: vLLM /v1/chat/completions, temperature=0, concurrency=16,
max_tokens=1024, chat_template_kwargs.enable_thinking=false, single
MI355X GPU (TP=1) for INT8 / TP=8 for BF16.
How to Use
With vLLM (Recommended)
vllm serve nameistoken/Qwen3.6-27B-Quark-W8A8-INT8 \
--tensor-parallel-size 1 \
--max-model-len 4096 \
--gpu-memory-utilization 0.9 \
--trust-remote-code
Chat completion call:
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "nameistoken/Qwen3.6-27B-Quark-W8A8-INT8",
"messages": [{"role": "user", "content": "Hello!"}],
"max_tokens": 256, "temperature": 0.7,
"chat_template_kwargs": {"enable_thinking": false}
}'
Hardware Requirements
- ~32 GB VRAM minimum (e.g., AMD MI300X / MI355X, NVIDIA A100-40G or larger).
Quantization Details
This model was quantized using AMD Quark's per-token per-channel INT8 scheme:
- Weight: INT8 per-channel symmetric static (
PerChannelMinMaxObserver,ch_axis=0). - Activation: INT8 per-token symmetric dynamic (
ch_axis=1). - Excluded layers:
lm_head,*embed_tokens*,*visual*,mtp*. - Export:
pack_method='reorder',weight_format='real_quantized',custom_mode='quark'. - Key-name post-process:
*.weight_quantizer.scale→*.weight_scale, drop*.weight_quantizer.zero_point(symmetric). Required for vLLMQuarkW8A8Int8path with transformers 5.x.
License
Apache License 2.0 (inherited from Qwen/Qwen3.6-27B). See LICENSE and NOTICE.