Qwen3-Omni-30B-A3B-Instruct — NVFP4 W4A4 (full thinker, awq_clip text calibration)
ModelOpt NVFP4 W4A4 quantization of Qwen/Qwen3-Omni-30B-A3B-Instruct.
The entire thinker text body — attention QKV/O + MoE experts — is quantized to NVFP4 with FP8 per-tensor input scales. Embeddings, norms, the MoE router (mlp.gate), lm_head, the audio encoder, the vision encoder, the talker, and code2wav stay in BF16.
| Total checkpoint size | 27.6 GiB (vs 66 GiB BF16) |
| Size reduction | ~58% |
| Hardware requirement | NVIDIA Blackwell (sm_100+) for native FP4 GEMM |
| Kernel path | FlashInfer Cutlass NvFp4 Linear + FlashInfer TRT-LLM NvFp4 MoE |
| Exported safetensors NaN bytes | 0 (calibrated with the ModelOpt-side fix; see Mitigations below) |
Accuracy — Daily-Omni (full split, n=1197)
Benchmarked on NVIDIA B200 (HBM3e, 183 GiB) via the in-tree
run_qwen_omni_acc_benchmark.py harness.
BF16 baseline run on the same hardware, same harness, same prompt set.
CUDA graphs enabled (no --enforce-eager).
| Model | Daily-Omni overall | n correct | Δ vs BF16 |
|---|---|---|---|
| Qwen/Qwen3-Omni-30B-A3B-Instruct (BF16) | 0.694 | 831 / 1197 | — |
| This checkpoint (W4A4 NVFP4) | 0.675 | 808 / 1197 | -1.9 pp |
Accuracy — OmniBench (n=200, via evalscope)
| Model | OmniBench mean_acc | Δ vs BF16 |
|---|---|---|
| BF16 | 0.455 | — |
| W4A4 NVFP4 | 0.450 | -0.5 pp |
Throughput / latency — vLLM serve on B200
Synthetic random workload (--random-input-len 256, --random-output-len 512, --num-prompts 200, no --enforce-eager, CUDA graphs captured for prefill + decode batch sizes 1..256).
| Concurrency | Metric | BF16 | W4A4 NVFP4 | W4A4 vs BF16 |
|---|---|---|---|---|
| 1 | Output tok/s | 156 | 150 | -4% |
| 1 | TPOT (ms) | 5.87 | 5.84 | tied |
| 1 | TTFT (ms) | 72 | 111 | +54% (BF16 wins prefill) |
| 8 | Output tok/s | 869 | 1002 | +15% |
| 8 | TPOT (ms) | 8.33 | 6.73 | -19% |
| 32 | Output tok/s | 2083 | 2766 | +33% |
| 32 | TPOT (ms) | 12.76 | 9.12 | -29% |
| 64 | Output tok/s | 3197 | 4112 | +29% |
| 64 | TPOT (ms) | 15.87 | 10.96 | -31% |
| 128 | Output tok/s | 4923 | 5665 | +15% |
| 128 | TPOT (ms) | 18.01 | 15.36 | -15% |
W4A4 wins on every metric at conc=8+. The single-stream TTFT loss reflects that FP4 prefill has small dequant overhead that BF16 native tensor cores don't pay; once concurrency saturates the GEMM, FP4 bandwidth savings dominate and W4A4 pulls ahead by 15-33% in tokens/s and 15-31% in per-token latency.
Stability
30-minute stress at concurrency=32: 3200 successful requests, 0 errors, 0 NaN collapses (regression guard on the !!!! failure mode that motivated vllm-omni#4025).
Calibration recipe
- Base model:
Qwen/Qwen3-Omni-30B-A3B-Instructinbfloat16. - ModelOpt:
nvidia-modelopt==0.44.0with the ModelOpt-side calibration fix applied (see Mitigations below). Exportedsafetensorscontain ModelOpt's raw output with no Python post-processing (--skip-post-export-nan-clamp). - Quant config:
mtq.NVFP4_DEFAULT_CFG+ algorithmawq_clip. - Calibration set: 1024 prompts from
HuggingFaceH4/ultrachat_200ktrain_sft, chat-templated through the Qwen3-Omni tokenizer, truncated to 512 tokens each. Text-only — multimodal samples were not used for this checkpoint. - Excluded patterns:
*audio_tower*,*visual*,*talker*,*code2wav*,*lm_head*,*mlp.gate*(router stays BF16 to avoid expert-routing drift). - Calibration time: ~2.4 h on a single RTX PRO 6000 Blackwell WS.
Inference
from vllm_omni import Omni
omni = Omni(model="YihongJin/Qwen3-Omni-30B-A3B-Instruct-NVFP4-W4A4-full-thinker-awqclip")
OpenAI-compatible server (recommended config for max throughput):
vllm serve YihongJin/Qwen3-Omni-30B-A3B-Instruct-NVFP4-W4A4-full-thinker-awqclip \
--omni --port 8000
Do not pass
--enforce-eagerfor benchmarks. CUDA graphs amortize kernel launch overhead and unlock the FP4 throughput wins above; with--enforce-eagerset, W4A4 TPOT degrades 10x relative to the CUDA-graph configuration.
Compute requirement: sm_100+ (Blackwell — B100, B200, RTX 5090, RTX Pro 6000) for the native FlashInfer FP4 GEMM kernel.
ModelOpt 0.44 NaN regression — two mitigation paths
ModelOpt 0.44's float32 -> torch.float8_e4m3fn cast of per-block weight_scale occasionally emits literal NaN bytes (E4M3 encoding 0x7F / 0xFF) when the pre-cast scale rounds above the FP8 max of 448 after the global-scale division. A single NaN byte in any weight_scale propagates through the FlashInfer FP4 GEMM into the residual stream and collapses the served model output to !!!!. Two complementary fixes:
-
Calibration-time (ModelOpt-side): clamp the pre-cast values to
torch.finfo(torch.float8_e4m3fn).maxbefore every.to(torch.float8_e4m3fn)at the two cast sites inmodelopt/torch/quantization/qtensor/nvfp4_tensor.pyandmodelopt/torch/export/quant_utils.py. This checkpoint was calibrated with that ModelOpt 0.44 patch applied — exported safetensors contain 0 NaN bytes. An upstream PR toNVIDIA/TensorRT-Model-Optimizeris in progress. -
Load-time (vllm-omni-side): vllm-project/vllm-omni#4025 installs a defensive override of
ModelOptNvFp4LinearMethod.process_weights_after_loadingthat scansweight_scalefor NaN bytes and clamps them to FP8 E4M3 max at worker init. Because this checkpoint is already clean, the override is a no-op safety net here; it primarily protects other in-the-wild W4A4 NVFP4 checkpoints that were exported with vanilla ModelOpt 0.44 and currently serve as!!!!. Self-extinguishes once vllm-omni's vllm pin includes the corresponding upstream vLLM fix; can be disabled withVLLM_OMNI_SKIP_NVFP4_NAN_CLAMP=1for diagnostics.
License
Apache-2.0 (inherits from the base Qwen3-Omni-30B-A3B-Instruct model).