Qwen3.5 27B NVFP4 MTP NVFP4 quantized Qwen/Qwen3.5 27B with the original bf16 MTP (Multi Token Prediction) head grafted back in for speculative decoding. Achieves a 3.3x speedup on a single DGX Spark over bf16 with TP=2 on two Sparks. Important flags language model only — Required. Qwen3.5 is natively multimodal; this skips the vision encoder (no vision weights are included). quantization modelopt — Required. Tells vLLM to use the NVFP4 weight format. num speculative tokens: 1 — The model has a single MTP layer. Setting this to 2 reuses the same layer twice with degraded acceptance rates. What's Different About This Quant Standard NVFP4 quantization via nvidia modelopt discards the MTP head because AutoModelForCausalLM doesn't load it. This checkpoint restores the MTP weights from the original model in bf16 and adds them to the quantization ignore list, giving you working speculative decoding out of the box. Quantization Details Property Value Method NVFP4 via nvidia modelopt 0.41.0 Group size 16 Calibration 256 samples from CNN/DailyMail (train split), max seq len=2048 Excluded from quantization lm head , all conv1d layers (Mamba blocks), all MTP linear layers MTP head bf16 (15 te…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy