Note: If you have a multi GPU SM120 Blackwell system (RTX 50/Pro), try my vLLM fork to resolve P2P / TP=2 issues (Pending PR into upstream). https://github.com/Gadflyii/vllm/tree/main GLM 4.7 Flash NVFP4 (Mixed Precision) This is a mixed precision NVFP4 quantization of zai org/GLM 4.7 Flash, a 30B A3B (30B total, 3B active) Mixture of Experts model. Quantization Strategy This model was made via custom quantization and calibration (128 samples, 2048 max seq len, neuralmagic/calibration, all 64 experts) scripts based on NVIDIA's approach for DeepSeek V3. It uses mixed precision to preserve accuracy: Component Precision Rationale MLP Experts FP4 (E2M1) 64 routed experts, 4 active per token Dense MLP FP4 (E2M1) First layer dense MLP Attention (MLA) BF16 Low rank compressed Q/KV projections are sensitive Norms, Gates, Embeddings BF16 Standard practice Performance Metric BF16 Uniform FP4 This Model MMLU Pro 24.83% 16.84% 23.55% Size 62.4 GB 18.9 GB 20.4 GB Compression 1x 3.3x 3.1x Accuracy Loss 8.0% 1.3% Usage Requirements vLLM : 0.14.0+ (for compressed tensors NVFP4 support) transformers : 5.0.0+ (for glm4 moe lite architecture) GPU : NVIDIA GPU with FP4 tensor core support (Blackwell,…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy