Microsoft Phi 4 4 bit AWQ Quantized Model (GEMM) This is a 4 bit AutoAWQ quantized version of Microsoft's Phi 4. It is optimized for fast inference using vLLM with minimal loss in accuracy. 🚀 Model Details Base Model: microsoft/phi 4 Quantization: 4 bit AWQ Quantization Method: AutoAWQ (Activation Aware Quantization) Group Size: 128 AWQ Version: GEMM Optimized Intended Use: Low VRAM inference on consumer GPUs VRAM Requirements: ✅ 8GB+ (Recommended) Compatibility: ✅ vLLM, Hugging Face Transformers (w/ AWQ support) 📌 How to Use in vLLM You can load this model directly in vLLM for efficient inference: Then, test it using cURL : 📌 How to Use in Python ( transformers + AWQ) To use this model with Hugging Face Transformers : 📌 Quantization Details This model was quantized using AutoAWQ with the following parameters: Bits: 4 bit quantization Zero Point Quantization: Enabled ( zero point=True ) Group Size: 128 ( q group size=128 ) Quantization Version: GEMM Method Used: AutoAWQ 📌 VRAM Requirements Model Size FP16 (No Quant) AWQ 4 bit Quantized Phi 4 14B ❌ Requires 20GB VRAM ✅ 8GB 12GB VRAM AWQ significantly reduces VRAM requirements , making it possible to run 14B models on consumer G…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy