INT4 google/gemma 3 12b it model Developed by: pytorch License: apache 2.0 Quantized from Model : google/gemma 3 12b it Quantization Method : INT4 Terms of Use : [Terms][terms] Inference with vLLM Install vllm nightly and torchao nightly to get some recent changes: Serving Then we can serve with the following command: Note: please use VLLM DISABLE COMPILE CACHE=1 to disable compile cache when running this code, e.g. VLLM DISABLE COMPILE CACHE=1 python example.py , since there are some issues with the composability of compile in vLLM and torchao, this is expected be resolved in pytorch 2.8. Inference with Transformers Install the required packages: Example: Quantization Recipe Install the required packages: Use the following code to get the quantized model: Note: to push to hub you need to run and use a token with write access, from https://huggingface.co/settings/tokens Model Quality We rely on lm evaluation harness to evaluate the quality of the quantized model. Here we only run on mmlu for sanity check. Benchmark google/gemma 3 12b it pytorch/gemma 3 12b it INT4 mmlu 71.51 68.96 chartqa (multimodal) 55.80 56.28 Reproduce Model Quality Results language eval Need to install lm eval…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy