Model Details This model is an int4 model with group size 128 and symmetric quantization of google/gemma 4 31B it generated by intel/auto round. Please follow the license of the original model. A youtube video by Fahd Mirza demonstrates how to use this model. Transformes inference ~~~python from transformers import AutoProcessor, AutoModelForCausalLM MODEL ID = "Intel/gemma 4 31B it int4 AutoRound" Load model processor = AutoProcessor.from pretrained(MODEL ID) model = AutoModelForCausalLM.from pretrained( MODEL ID, dtype="auto", device map="auto" ) Prompt add image before text messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://raw.githubusercontent.com/google gemma/cookbook/refs/heads/main/Demos/sample data/GoldenGate.png"}, {"type": "text", "text": "What is shown in this image?"} ] } ] Process input inputs = processor.apply chat template( messages, tokenize=True, return dict=True, return tensors="pt", add generation prompt=True, ).to(model.device) input len = inputs["input ids"].shape[ 1] Generate output outputs = model.generate( inputs, max new tokens=512) response = processor.decode(outputs[0][input len:], skip special tokens=False) Parse output print(…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy