Gemma 4 26B A4B MoE IT QAT Assistant — INT4 W4A16 Compressed-Tensors (Marlin)

This is the MTP draft/assistant model for Gemma 4 26B A4B MoE, quantized from google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant (BF16) to INT4 W4A16 compressed-tensors format with group_size=128, optimized for Marlin kernel acceleration.

It is designed to be used alongside NeoChen1024/gemma-4-26B-A4B-it-qat-W4A16 as the main model with SGLang's Frozen-KV MTP speculative decoding.

Why This Model?

Using the original BF16 assistant model with MTP speculative decoding causes a severe short-sequence regression on Ampere GPUs when the main model uses Marlin INT4 kernels. This INT4 quantized draft model uses the same Marlin kernels, eliminating the regression and delivering a significant speedup:

26% faster on short sequences vs no-MTP baseline. Comparable on long sequences.

Benchmarks on NVIDIA RTX 3090 (24 GB) with SGLang 0.5.12, mem-fraction-static=0.85, cuda-graph-max-bs=1, context-length=4096, speculative-num-steps=3, speculative-num-draft-tokens=4.

How It Was Created

Standard quantization tools (llmcompressor, compressed_tensors, bitsandbytes) all fail on the Gemma4AssistantForCausalLM architecture due to its shared_kv_states forward signature and FX tracer incompatibility.

Instead, this model was quantized with manual RTN (Round-to-Nearest) symmetric INT4 quantization, producing output in the exact compressed-tensors pack-quantized format that SGLang's loader expects:

• Method: Per-group symmetric RTN, scale = max(|W_group|) / 7
• Group size: 128 (Marlin kernel sweet spot, vs 32 in Google's QAT main model)
• Format: compressed-tensors pack-quantized (weight_packed + weight_scale + weight_shape)
• Packing: 8 × 4-bit values per int32, standard compressed-tensors layout
• Ignored layers: lm_head, embed_tokens (kept in BF16)
• Tied weights: lm_head.weight = model.embed_tokens.weight (same as original)

23 Linear layers quantized across the 4-layer assistant transformer, including pre_projection (1024→5632) and post_projection (2816→1024).

Running with SGLang

Important: The 26B MoE uses GELU activation (not SiLU). You need the GELU MoE patch from PR sgl-project/sglang#24280. Save the patch script below as /tmp/patch_gelu_moe_v2.sh.

docker run -d \
  --name sglang-26b-mtp \
  --gpus all \
  --network host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v /tmp/patch_gelu_moe_v2.sh:/patch.sh:ro \
  lmsysorg/sglang:gemma4-mtp \
  bash -c "bash /patch.sh && \
  pip install --no-deps --quiet 'git+https://github.com/huggingface/transformers.git@1423d22f7a3b62e8c70ad67b58ec25cd9b675897' 2>&1 && \
  python3 -m sglang.launch_server \
    --model-path NeoChen1024/gemma-4-26B-A4B-it-qat-W4A16 \
    --quantization compressed-tensors \
    --kv-cache-dtype fp8_e5m2 \
    --mem-fraction-static 0.85 \
    --context-length 4096 \
    --cuda-graph-max-bs 1 \
    --speculative-algorithm FROZEN_KV_MTP \
    --speculative-num-steps 3 \
    --speculative-num-draft-tokens 4 \
    --speculative-eagle-topk 1 \
    --speculative-draft-model-path kunhunjon/gemma-4-26B-A4B-it-qat-assistant-w4a16-ct \
    --speculative-draft-model-quantization compressed-tensors \
    --max-running-requests 2 \
    --reasoning-parser gemma4 \
    --tool-call-parser gemma4 \
    --dtype bfloat16 \
    --trust-remote-code \
    --host 0.0.0.0 --port 30000"

Notes:

• --speculative-eagle-topk 1 is required even for FROZEN_KV_MTP due to an SGLang bug (it unconditionally multiplies eagle_topk * num_steps).
• --mem-fraction-static 0.85 balances KV cache capacity vs CUDA graph memory. See the Memory Tuning section below for alternatives.
• --cuda-graph-max-bs 1 is required to fit everything in VRAM on 24 GB GPUs.
• --kv-cache-dtype fp8_e5m2 uses FP8 KV cache (supported on Ampere and later). Use fp8_e4m3 on Hopper+ or omit for BF16 KV cache.
• The transformers git commit is needed for gemma4_assistant architecture support.

MTP Acceptance Rates

Settings: speculative_num_steps=3, speculative_num_draft_tokens=4.

Memory Tuning

The --mem-fraction-static parameter controls the KV cache pool size. Higher values give more KV capacity but less room for CUDA graphs and the draft model:

0.85 is recommended — it doubles KV capacity vs 0.80 while maintaining performance. Note: --context-length does NOT affect memory; the pool size is set solely by --mem-fraction-static.

VRAM Breakdown

Hugging Face | GitHub | Launch Blog | Documentation
License: Apache 2.0 | Authors: Google DeepMind

[!Note] This model card is for the new versions of the Gemma 4 family optimized with Quantization-Aware Training (QAT), which allows preserving similar quality to bfloat16 while dramatically reducing the memory requirements to load the model. Four versions of the QAT checkpoints are available:

• Unquantized QAT checkpoints (Q4_0): Half-precision weights extracted from the QAT pipeline, ideal for custom downstream compilation and research. Available for Gemma 4 E2B, E4B, 12B, 26B A4B, and 31B, and their drafter models.
• GGUF (Q4_0): Ready-to-deploy formats for broad ecosystem compatibility. Available for Gemma 4 E2B, E4B, 12B, 26B A4B, and 31B.
• Mobile-optimized (wNa8o8): A custom schema engineered explicitly for mobile hardware efficiency. It features targeted 2-bit decoding layers, optimized KV caches, and static activations to maximize VRAM savings. Available for Gemma 4 E2B and E4B.
• Compressed Tensors (w4a16): QAT checkpoints serialized in the compressed-tensors format for native, optimized inference with vLLM. Available for Gemma 4 E2B, E4B, 12B, and 31B.

Gemma is a family of open models built by Google DeepMind. Gemma 4 models are multimodal, handling text and image input (with audio supported on E2B, E4B, and 12B) and generating text output. This release includes open-weights models in both pre-trained and instruction-tuned variants. Gemma 4 features a context window of up to 256K tokens and maintains multilingual support in over 140 languages.

Featuring both Dense and Mixture-of-Experts (MoE) architectures, Gemma 4 is well-suited for tasks like text generation, coding, and reasoning. The models are available in five distinct sizes: E2B, E4B, 12B, 26B A4B, and 31B. Their diverse sizes make them deployable in environments ranging from high-end phones to laptops and servers, democratizing access to state-of-the-art AI.

Gemma 4 introduces key capability and architectural advancements:

• Reasoning – All models in the family are designed as highly capable reasoners, with configurable thinking modes.

• Extended Multimodalities – Processes Text, Image with variable aspect ratio and resolution support (all models), Video, and Audio (featured natively on the E2B, E4B, and 12B models).

• Diverse & Efficient Architectures – Offers Dense and Mixture-of-Experts (MoE) variants of different sizes for scalable deployment.

• Optimized for On-Device – Smaller models are specifically designed for efficient local execution on laptops and mobile devices.

• Increased Context Window – The small models feature a 128K context window, while the medium models support 256K.

• Enhanced Coding & Agentic Capabilities – Achieves notable improvements in coding benchmarks alongside native function-calling support, powering highly capable autonomous agents.

• Native System Prompt Support – Gemma 4 introduces native support for the system role, enabling more structured and controllable conversations.

Models Overview

Gemma 4 models are designed to deliver frontier-level performance at each size, targeting deployment scenarios from mobile and edge devices (E2B, E4B) to consumer GPUs and workstations (12B, 26B A4B, 31B). They are well-suited for reasoning, agentic workflows, coding, and multimodal understanding.

The models employ a hybrid attention mechanism that interleaves local sliding window attention with full global attention, ensuring the final layer is always global. This hybrid design delivers the processing speed and low memory footprint of a lightweight model without sacrificing the deep awareness required for complex, long-context tasks. To optimize memory for long contexts, global layers feature unified Keys and Values, and apply Proportional RoPE (p-RoPE).

Dense Models

Mixture-of-Experts (MoE) Model

Best Practices

1. Sampling Parameters

Use the following standardized sampling configuration across all use cases:

• temperature=1.0
• top_p=0.95
• top_k=64

2. Thinking Mode Configuration

To properly manage the thinking process, use the following control tokens:

• Trigger Thinking: Thinking is enabled by including the |<<|think|> token at the start of the system prompt. To disable thinking, remove the token.
• Standard Generation: When thinking is enabled, the model will output its internal reasoning followed by the final answer using this structure: |<<|channel>thought\n[Internal reasoning]<<channel|>

3. Multi-Turn Conversations

• No Thinking Content in History: In multi-turn conversations, the historical model output should only include the final response. Thoughts from previous model turns must not be added before the next user turn begins.

Ethics and Safety

As open models become central to enterprise infrastructure, provenance and security are paramount. Developed by Google DeepMind, Gemma 4 undergoes the same rigorous safety evaluations as our proprietary Gemini models.

Evaluation Approach

Gemma 4 models were developed in partnership with internal safety and responsible AI teams. A range of automated as well as human evaluations were conducted to help improve model safety. These evaluations align with Google's AI principles, as well as safety policies.

Ethical Considerations and Risks

• Bias and Fairness – VLMs trained on large-scale data can reflect socio-cultural biases. Gemma 4 underwent careful scrutiny and evaluations to mitigate bias risks.
• Misinformation and Misuse – Guidelines are provided for responsible use. See the Responsible Generative AI Toolkit.
• Transparency and Accountability – This model card summarizes details on architecture, capabilities, limitations, and evaluation processes.