AWQ INT4 google/gemma 3 27b it model Developed by: pytorch License: apache 2.0 Quantized from Model : google/gemma 3 27b it Quantization Method : AWQ INT4 Terms of Use : [Terms][terms] Calibrated with 30 samples of mmlu philosophy , got eval accuracy of 80.06, while gemma 3 27b it INT4 is 77.17, and bfloat16 baseline is 79.42 Inference with vLLM Install vllm nightly and torchao nightly to get some recent changes: Serving Then we can serve with the following command: Note: please use VLLM DISABLE COMPILE CACHE=1 to disable compile cache when running this code, e.g. VLLM DISABLE COMPILE CACHE=1 python example.py , since there are some issues with the composability of compile in vLLM and torchao, this is expected be resolved in pytorch 2.8. Inference with Transformers Install the required packages: Example: Quantization Recipe Install the required packages: Use the following code to get the quantized model: Note: to push to hub you need to run and use a token with write access, from https://huggingface.co/settings/tokens Model Quality We rely on lm evaluation harness to evaluate the quality of the quantized model. Here we only run on mmlu for sanity check. Benchmark google/gemma 3 27b…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy