Overview This model is optimized for use with VLLM on NVIDIA GPUs with compute capability 8.0 (Ampere, A100, A10, 3090, etc.). It utilizes a weight only FP8 Marlin kernel, providing an efficient W8A16 configuration. Key Features of FP8 Marlin The NeuralMagic FP8 Marlin kernel achieves impressive efficiency by packing 4 8 bit values into an int32 and performing a 4xFP8 to 4xFP16/BF16 dequantization using bit arithmetic and SIMT operations. This approach yields nearly a 2x speedup over FP16 on most models while maintaining near lossless quality . FP8 Advantages on NVIDIA GPUs On newer NVIDIA GPUs (4090/H100 or later), dedicated FP8 tensor cores and hardware allow fast conversion from FP8 to BF16/FP16, maximizing performance. However, older GPUs lack this specific hardware support, preventing activation quantization if we want to leverage FP8. The Marlin kernel addresses this gap effectively, enabling performance gains on Ampere cards (e.g., 3090, A100) without needing full tensor core support. Traditional int8 quantization methods often require extensive overhead for data type conversion between int8 and fp16, making them less efficient for inference. Marlin’s FP8 kernel bypasses thi…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy