Phi4 mini model quantized with torchao float8 dynamic activation and float8 weight quantization (per row granularity), by PyTorch team. Use it directly, or serve using vLLM with 36% VRAM reduction (5.70 GB needed), 1.15x 1.2x speedup and little to no accuracy impact on H100. Inference with vLLM Install vllm nightly to get some recent changes: To use fbgemm kernels: Code Example Note: please use VLLM DISABLE COMPILE CACHE=1 to disable compile cache when running this code, e.g. VLLM DISABLE COMPILE CACHE=1 python example.py , since there are some issues with the composability of compile in vLLM and torchao, this is expected be resolved in pytorch 2.8. Serving Then we can serve with the following command: Inference with Transformers Install the required packages: To use fbgemm kernels: Example: Quantization Recipe Install the required packages: Use the following code to get the quantized model: Note: to push to hub you need to run and use a token with write access, from https://huggingface.co/settings/tokens Model Quality We rely on lm evaluation harness to evaluate the quality of the quantized model. Benchmark Phi 4 mini ins Phi 4 mini instruct FP8 Popular aggregated benchmark mmlu (…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy