Gemma 4 12B-it GGUF — Quantized by BatiAI

Optimized GGUF quantizations of google/gemma-4-12B-it — Google DeepMind's encoder-free multimodal model (text + image + audio + video) that runs on a 16 GB Mac. Built directly from official Google BF16 weights by BatiAI for BatiFlow.

Why Gemma 4 12B?

• 26B-MoE-class quality at <half the memory — Google's release notes put the 12B near the Gemma 4 26B MoE on standard benchmarks
• Encoder-free multimodal — first mid-sized model with native audio. Raw image patches and audio waveforms project directly into the LLM via lightweight linear layers — no separate vision/audio encoder
• Strong vision + reasoning: DocVQA 94.9 · InfoVQA 88.4 · MMMU-Pro 69.1 · AIME 2026 77.5 · MATH-Vision 79.7
• 256K context, 140+ languages
• Apache 2.0 — commercial-friendly
• Released June 3, 2026

Quick Start

ollama pull batiai/gemma4-12b:q4

Available Quantizations

IQ variants use imatrix calibration. Q2/Q3 (8GB Macs) are not in Google's reference set — BatiAI adds them.

Two modes — text-only by default, multimodal opt-in

Upstream Gemma 4 12B-it is multimodal. In the GGUF ecosystem this is delivered as two files: a main model GGUF (the LLM) and a small mmproj GGUF holding the vision + audio projectors together (encoder-free → the projector is tiny, ~167 MB, and one BF16 mmproj works with every quant).

What actually works locally (verified on M4 Max 128GB):

⚠️ Ollama can't do images/audio for Gemma 4 yet (0.20 doesn't know the gemma4uv/gemma4ua projectors). Multimodal needs llama-server built from a recent llama.cpp master that includes the Gemma 4 projectors (the gemma4v/gemma4uv/gemma4a/gemma4ua clip graphs). Older builds fail with unknown projector type: gemma4uv.

# Multimodal via llama-server (recent llama.cpp master + --jinja required)
llama-server -m gemma-4-12B-it-Q4_K_M.gguf \
  --mmproj mmproj-google-gemma-4-12B-it-BF16.gguf \
  --host 127.0.0.1 --port 8899 -ngl 99 -c 8192 --jinja
# → GET /props returns {"vision": true, "audio": true}; POST /v1/chat/completions with image_url / input_audio

Gemma 4 12B is a reasoning model — give it enough max_tokens (image descriptions emit 700+ tokens incl. a thinking block; the answer arrives in reasoning_content + content).

RAM Requirements

Why BatiAI Quantization?

Technical Details

• Original Model: google/gemma-4-12B-it
• Architecture: gemma4_unified — encoder-free unified multimodal, 12B dense, 48 layers, 256K context
• Modalities: text + image + audio (≤30s) + video (≤60s)
• License: Apache 2.0
• Quantized with: llama.cpp + imatrix
• Quantized by: BatiAI

File checksums (SHA-256)

shasum -a 256 google-gemma-4-12B-it-Q4_K_M.gguf   # verify after download

About BatiFlow

BatiFlow — free, on-device AI automation for Mac. 5MB app, 100% local, unlimited.

License

Quantized from google/gemma-4-12B-it. License: Apache 2.0.

Benchmarks

Text (ollama run --verbose, thinking ON default):

Multimodal (llama-server static commit 308f61c + Q4_K_M + mmproj, M4 Max 128GB — measured):

/props → {"vision": true, "audio": true}. Gemma 4 is a reasoning model — responses include a <think> block, so give generous max_tokens. Video not supported (no gemma4 video projector in llama.cpp).