Note: You must run with disable shared experts fusion in sglang, otherwise it will incorrectly attempt to fuse the BF16 shared expert. Update 5/6/26 sglang container updated with SM120/RTX 6000 Native Sparse Attention support Model Description GLM 5.1 NVFP4 is an NVFP4 quantized version of zai org/GLM 5.1, a 744B parameter Mixture of Experts language model with 40B active parameters, 256 experts per MoE layer (8 activated per token), and DeepSeek Sparse Attention (DSA). Quantized directly from the full BF16 checkpoint (zai org/GLM 5.1), not the FP8 release , to NVFP4 (4 bit with blockwise FP8 scales per 16 elements) using NVIDIA Model Optimizer. What's quantized Only the non shared MoE expert MLP projections are quantized to NVFP4. Attention weights are left in BF16, in addition to the dense MLPs (layers 0 3) and the shared experts. Since the MoE expert weights constitute the vast majority of model parameters in an MoE architecture, this still yields significant memory savings. Calibration uses natural top k routing rather than forcing all experts to activate, so each expert's quantization scales reflect the token distributions it actually sees during inference. To compensate, cali…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy