Qwen3 Embedding 8B GPTQ (4 bit Quantized) GPTQ W4A16 quantized version of Qwen3 Embedding 8B Original Size: 15GB → Quantized Size: 4.5GB ( 70% reduction ) Semantic Accuracy: 99.14% preserved Context Window: Up to 32k tokens Quantization: GPTQ W4A16 (4 bit weights, 16 bit activations) Model Details Base Model: Qwen/Qwen3 Embedding 8B Parameters: 8B Embedding Dim: 3072 Max Sequence Length: 32768 Quantization: GPTQ (4 bit weights, 16 bit activations) Calibration Samples: 128 Dataset: ultrachat 200k Performance Metric Original Quantized Preservation Model Size 15GB 4.5GB 70% reduction Semantic Accuracy 0.824 0.827 99.14% Max Difference 0.019 1.9% Test Results Comprehensive testing with 11 test pairs: Mean Absolute Difference: 0.007 (0.86%) Maximum Difference: 0.019 (1.9%) Rating: ✅ EXCELLENT ( 95% preserved) Usage vLLM (Recommended) Python OpenAI compatible API Quantization Details Method: GPTQ (Group wise Post Training Quantization) Configuration: Weights: 4 bit Activations: 16 bit Group Size: 128 Scheme: W4A16 Damping Factor: Default (auto) Why GPTQ over AWQ? AWQ calibration produced NaN values for Qwen3 Embedding 8B GPTQ more stable for embedding models GPTQ provides better accuracy…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy