Qwopus3.6-27B-v2-NVFP4

Parameters: 27B (base model)
Quantized to NVFP4 Vision Encoder + MTP kept in BF16

Base Model: Jackrong/Qwopus3.6-27B-v2

Overview

This repository contains a quantized version of Qwopus3.6-27B-v2 (based on the Qwen3.5 architecture). The model has been quantized from BF16 to NVFP4 format using the NVIDIA TensorRT Model Optimizer (modelopt).

To maintain high performance and accuracy, a hybrid precision strategy was applied: the core LLM weights are quantized to 4-bit, while the Vision Encoder and Multi-Token Prediction (MTP) modules are strictly preserved in their native BF16 precision.

The resulting checkpoint is fully compatible with vLLM for high-throughput, low-latency inference.

Hardware Recommendation

• Target Hardware: Highly recommended for NVIDIA Blackwell architecture GPUs (e.g., NVIDIA RTX 50 series, RTX PRO series, and GB100/GB10 chips) which feature native hardware support for FP4.

Model Specifications

• Original Size: ~54 GB (BF16)
• Quantized Size: ~18 GB (~0.33x compression ratio)
• Format: Hugging Face safetensors (compatible with vLLM via --quantization modelopt)
• Preserved Modules: Vision tower and MTP heads remain in BF16 to prevent degradation in image understanding and speculative decoding efficiency.

Key Design Decisions & Rationale

Quantization Pipeline (7 Steps)

The quantization process followed a strict 7-step engineering pipeline:

1. Load Model: Loaded the full Qwen3_5ForConditionalGeneration model in BF16 onto the GPU.
2. Prepare Calibration Data: Sampled 256 conversation sequences from neuralmagic/calibration, tokenized with the official chat template (Max length: 2,048 tokens).
3. Configure Quantization Rules: Extended NVFP4_DEFAULT_CFG with explicit disable rules for *visual* (vision encoder) and *mtp* (multi-token prediction heads).
4. Quantize & Calibrate: Executed mtq.quantize() with a forward loop across all 256 calibration samples, collecting per-layer activation statistics (amax values) to compute static FP4 scaling factors.
5. Export Checkpoint: Serialized quantized weights using export_hf_checkpoint(), outputting FP4 weights and float8 input_scales.
6. Graft MTP Weights: Extracted all mtp.* tensors from the original BF16 model and wrote them into a separate shard (model_mtp.safetensors). Updated model.safetensors.index.json accordingly to prevent loading/saving artifacts.
7. Patch Config: Registered all MTP module names under quantization_config.ignore in config.json, instructing vLLM to automatically bypass quantization and load these layers in BF16.

Deployment via vLLM

You can serve this model using vLLM with the following command:

python3 -m vllm.entrypoints.openai.api_server \
    --model /path/to/Qwopus3.6-27B-v2-NVFP4 \
    --served-model-name Qwopus3.6-27B-v2-NVFP4 \
    --quantization modelopt \
    --trust-remote-code \
    --dtype auto \
    --gpu-memory-utilization 0.93 \
    --max-model-len 112000 \
    --max-num-seqs 4 \
    --max-num-batched-tokens 8192 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --default-chat-template-kwargs '{"enable_thinking": true}' \
    --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \
    --reasoning-parser qwen3 \
    --host 0.0.0.0 \
    --port 8000