Qwen3.6-35B-A3B-Hybrid-INT4-FP8-MTP

Hybrid quantization of Qwen/Qwen3.6-35B-A3B optimized for single-GPU deployment on the NVIDIA DGX Spark / ASUS GX10 (GB10, 128 GB unified memory).

• MoE expert FFN layers → INT4 (Intel AutoRound group_size=128)
• Attention, shared experts, LM head, embeddings → FP8 (E4M3 blockwise 128×128, calibrated by Qwen)
• MTP (Multi-Token-Prediction) drafter weights included for speculative decoding

The result is a ~20 GB checkpoint (vs. ~70 GB BF16, ~35 GB FP8) that runs at ~92–103 tok/s single-request and ~164 tok/s aggregate at 16× concurrency on a single GB10, while keeping headroom for 256k context KV cache.

Headline numbers

Measured on ASUS GX10 / DGX Spark (GB10, 128 GB unified memory) with the patched vLLM runtime described below, single request unless stated otherwise.

For reference, phuongncn's README reports for the same hardware:

Quantization strategy — why hybrid?

LLM decode is memory-bandwidth-bound. The smaller the weights, the faster. But not all layers tolerate aggressive quantization equally:

This split puts ~90% of the forward-pass bandwidth on the fast INT4 path, while the quality-critical 10% stays on the native GB10 FP8 tensor-core path.

INT4 alone would visibly hurt attention quality; FP8 alone would use ~15 GB more memory that we'd rather give to the KV cache at 256k context.

→ Net: ~3–4× speedup over BF16, ~2× over FP8 native, with quality close to FP8 native.

MTP — Multi-Token-Prediction

This checkpoint includes MTP drafter weights (model_extra_tensors.safetensors). The model has exactly one MTP hidden layer (text_config.mtp_num_hidden_layers = 1).

When loaded with vLLM --speculative-config '{"method":"mtp","num_speculative_tokens":N}', the same drafter layer is invoked recursively N times. Empirically optimal value: N=2.

Per-position acceptance rate (measured, vLLM's SpecDecoding metrics):

Recommendation: num_speculative_tokens=2. =3 is occasionally faster on peak but doubles latency variance. =5 is slower than MTP-off because wasted compute on positions 4–5 outweighs gains.

Quick start (vLLM, DGX Spark)

A pre-built Docker image with the patched vLLM runtime is published on GHCR — no build steps required.

# 1) Download model weights (~20 GB)
hf download kleybrink/Qwen3.6-35B-A3B-Hybrid-INT4-FP8-MTP \
  --local-dir ~/models/qwen36-35b-hybrid-int4fp8-mtp

# 2) Start the server
docker run -d \
  --name vllm-qwen36-hybrid \
  --gpus all --net=host --ipc=host --shm-size=16g \
  -v ~/models:/local_models \
  ghcr.io/kleybrink/vllm-qwen36-hybrid:latest \
  serve --model /local_models/qwen36-35b-hybrid-int4fp8-mtp \
  --served-model-name qwen3.6-35b-hybrid \
  --port 8000 --host 0.0.0.0 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.8 \
  --tensor-parallel-size 1 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --trust-remote-code \
  --enable-prefix-caching \
  --quantization autoround \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
  --load-format fastsafetensors

Available image tags (ghcr.io/kleybrink/vllm-qwen36-hybrid):

Test:

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.6-35b-hybrid",
    "messages": [{"role":"user","content":"Hello!"}]
  }'

Thinking mode

By default this image runs with enable_thinking=false for raw speed. Per-request override:

{ "chat_template_kwargs": { "enable_thinking": true } }

Vision

The base model is multimodal. Send images via base64 data URLs:

{
  "messages": [{
    "role": "user",
    "content": [
      {"type": "text", "text": "Describe this image."},
      {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}
    ]
  }]
}

Note: the User-Agent vLLM uses for fetching images is rejected by some CDNs (e.g. Wikimedia thumbnails). Prefer base64 data URLs or your own static hosting.

Hardware requirements

• NVIDIA DGX Spark / ASUS GX10 (GB10) — 128 GB unified memory. This release is built and tested only on this hardware.
• The checkpoint requires the patched vLLM runtime that ships in ghcr.io/kleybrink/vllm-qwen36-hybrid — no manual setup needed. The build context and patches live in the upstream phuongncn/asus-gx10-qwen35-speed-hack repo.

What's in the box

qwen36-35b-hybrid-int4fp8-mtp/
├── README.md                                     ← this file
├── LICENSE                                       ← Apache-2.0
├── NOTICE                                        ← attributions and upstream sources
├── .gitattributes                                ← LFS tracking for weights
├── config.json                                   ← model config + _hybrid_quant_info marker
├── quantization_config.json                      ← AutoRound INT4 + FP8 hybrid spec
├── generation_config.json
├── chat_template.jinja
├── preprocessor_config.json                      ← vision preprocessor
├── processor_config.json
├── tokenizer.json
├── tokenizer_config.json
├── model.safetensors.index.json
├── model-{00001..00010}-of-00010.safetensors     ← merged INT4+FP8 weights
└── model_extra_tensors.safetensors               ← MTP drafter layer

Hybrid quant info (from config.json)

{
  "description": "Hybrid AutoRound INT4 + FP8 checkpoint for single-GPU deployment",
  "moe_experts": "AutoRound INT4",
  "dense_layers": "FP8 E4M3 block weights from Qwen/Qwen3.6-35B-A3B-FP8",
  "source_int4": "Intel/Qwen3.6-35B-A3B-int4-AutoRound",
  "source_fp8": "Qwen/Qwen3.6-35B-A3B-FP8",
  "converter": "build-hybrid-checkpoint.py",
  "promoted_quantized_dense_layers": 250
}

Credits

This checkpoint stands on the shoulders of others. Please credit them too.

• Qwen Team / Alibaba Cloud — base model architecture, weights, FP8 calibrated release, tokenizer, chat template. (Qwen/Qwen3.6-35B-A3B and Qwen/Qwen3.6-35B-A3B-FP8)
• Intel — AutoRound INT4 expert weights, calibration-aware quantizer that makes aggressive 4-bit quantization on MoE experts viable. (Intel/Qwen3.6-35B-A3B-int4-AutoRound, see also intel/auto-round)
• albond — original Hybrid INT4+FP8 merging technique and patched vLLM Docker image, originally for Qwen3.5-122B. (albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4)
• phuongncn (Fususu) — generalized albond's technique to Qwen3.5-35B and Qwen3.6-35B, added MTP integration and Qwen3.6 runtime patches, packaged the build pipeline. (phuongncn/asus-gx10-qwen35-speed-hack)
• vLLM project — inference runtime.

This Qwen3.6 hybrid checkpoint was built and benchmarked on a DGX Spark by Dandy Kleybrink. The empirically optimal num_speculative_tokens=2 was determined by sweep on this hardware.

Citation

If you use this checkpoint, please cite the underlying work:

@misc{qwen3_6,
  title  = {Qwen3.6 Technical Report},
  author = {Qwen Team},
  year   = {2026},
  url    = {https://qwenlm.github.io/}
}

@article{cheng2023optimize,
  title   = {Optimize weight rounding via signed gradient descent for the quantization of LLMs},
  author  = {Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi},
  journal = {arXiv preprint arXiv:2309.05516},
  year    = {2023}
}

License

Apache-2.0 — same as the upstream base model. This redistribution combines weights from multiple Apache-2.0 sources; their licensing terms apply to the corresponding portions. See NOTICE for the full attribution chain.

Disclaimers

Model behavior

• The model can produce factually incorrect, biased, or otherwise inappropriate output. Apply your own safety review before deploying in user-facing applications.
• INT4 quantization on the expert FFN may introduce subtle quality degradation relative to native FP8 — use Qwen/Qwen3.6-35B-A3B-FP8 directly if maximum quality is your priority and you have the VRAM headroom.

Performance numbers

The throughput figures in this card are measurements under the specific conditions described, not guarantees. They were obtained on a single ASUS GX10 / DGX Spark (GB10, 128 GB unified memory) with the patched vLLM runtime and the configuration listed in the Quick start section, using uvx llama-benchy 0.3.7. Numbers will differ on different hardware, drivers, kernel versions, or workload shapes (longer contexts, image input, alternate sampling configs).

Warranty and liability

This redistribution is provided "AS IS", without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose, and noninfringement. In no event shall the redistributor be liable for any claim, damages, or other liability — whether in an action of contract, tort, or otherwise — arising from, out of, or in connection with the model or its use.

This is a redistribution of derived weights from open-weight models released under permissive licenses; see LICENSE (Apache-2.0) and NOTICE for the full chain. Apache-2.0 § 7 (Disclaimer of Warranty) and § 8 (Limitation of Liability) apply in full.

Affiliations

The redistributor is not affiliated with Alibaba Cloud, the Qwen team, Intel Corporation, the vLLM project, or any of the upstream tooling authors mentioned in this card or in NOTICE. Trademarks belong to their respective owners. References to upstream projects are made for the sole purpose of attribution and reproducibility.