DeepSeek R1 AWQ AWQ of DeepSeek R1. Quantized by Eric Hartford and v2ray. This quant modified some of the model code to fix an overflow issue when using float16. To serve using vLLM with 8x 80GB GPUs, use the following command: You can download the wheel I built for PyTorch 2.6, Python 3.12 by clicking here, the benchmark below was done with this wheel, it contains 2 PR merges and an unoptimized FlashMLA (still faster than Triton) for A100 which boosted performance a lot. The vLLM repo which contained A100 FlashMLA can be found at LagPixelLOL/vllm@sm80 flashmla, which is a fork of vllm project/vllm. The A100 FlashMLA it used is based on LagPixelLOL/FlashMLA@vllm, which is a fork of pzhao eng/FlashMLA. TPS Per Request GPU \ Batch Input Output B: 1 I: 2 O: 2K B: 32 I: 4K O: 256 B: 1 I: 63K O: 2K Prefill : : : : : : : : : : 8x H100/H200 61.5 30.1 54.3 4732.2 4x H200 58.4 19.8 53.7 2653.1 8x A100 80GB 46.8 12.8 30.4 2442.4 8x L40S 46.3 OOM OOM 688.5 Note: The A100 config uses an unoptimized FlashMLA implementation, which is only superior than Triton during high context inference, it would be faster if it's optimized. The L40S config doesn't support FlashMLA, thus the Triton implementat…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy