gemma-4-26B-A4B-it-qat-W4A16

Lossless Q4_0 GGUF → compressed-tensors pack-quantized W4A16 model for vLLM deployment.

What This Is

• Base model: google/gemma-4-26B-A4B-it
• Source format: Q4_0 GGUF (QAT-trained, exported by llama.cpp)
• Target format: compressed-tensors pack-quantized
• Quantization: 4-bit symmetric integer, group-wise (group_size=32)
• Scale dtype: bfloat16 (or float16 via --dtype)
• Marlin padding: MoE and Dense MLP blocks padded to min_thread_k=128 (Dense MLP skippable via --skip-dense-mlp)
• Vision tower: copied from --unquantized-ref source (unquantized bfloat16)
• k_eq_v handling: v_proj absent for full_attention layers (5, 11, 17, 23, 29)

Conversion

python convert_gguf_q4_0_to_ct.py \
    --gguf-path gemma-4-26B_q4_0-it.gguf \
    --unquantized-ref gemma-4-26B-A4B-it-qat-q4_0-unquantized \
    --output-dir gemma-4-26B-A4B-it-qat-W4A16

The conversion is lossless at the nibble level — every Q4_0 scale and quantized weight value is preserved bit-for-bit from the GGUF source.

Arguments

Marlin Padding

The vLLM Marlin kernel requires in_features % 128 == 0. The script automatically pads affected FFN blocks with zero-fill rows/columns:

Padded regions use q=8 (q_signed=0 → zero weight). The config.json is updated with padded intermediate_size=2176 and moe_intermediate_size=768.

Tensor Parallel (TP > 1)

Dense MLP with Marlin fails with TP≥2 because 2176/2 = 1088 is still not divisible by min_thread_k=128. MoE experts work fine (768/2 = 384, 384/128 = 3).

Use --skip-dense-mlp to keep Dense MLP unquantized for TP>1 deployments (adds ~725 MiB VRAM):

python convert_gguf_q4_0_to_ct.py \
    --gguf-path gemma-4-26B_q4_0-it.gguf \
    --unquantized-ref gemma-4-26B-A4B-it-qat-q4_0-unquantized \
    --output-dir gemma-4-26B-A4B-it-qat-W4A16 \
    --skip-dense-mlp

vLLM Serving

vllm serve gemma-4-26B-A4B-it-qat-W4A16 \
    --gpu-memory-utilization 0.85 \
    --enforce-eager \
    --trust-remote-code

Important: Use /v1/chat/completions (not /v1/completions) — Gemma4 is an instruct model that requires the chat template.

Tested with vLLM v0.22.1, CUDA compute capability 8.9 (RTX 4080 Super). Model loads at ~16.4 GiB GPU memory.

Key Design Decisions

k_eq_v (full_attention) Layers

vLLM duplicates k_proj as v_proj at runtime for full_attention layers. The checkpoint must NOT contain real v_proj tensors for these layers. The GGUF source already has no v_proj for full_attention layers (5, 11, 17, 23, 29), so this is handled naturally.

Vision Tower

GGUF does not contain vision tower weights. They are copied as unquantized bfloat16 from the --unquantized-ref source model (alongside all other non-Q4_0 tensors). Supports both single-file and sharded (index.json) models.

Non-Q4_0 Tensors

All non-quantized tensors — embed_tokens, layer norms, router weights, router.scale, layer_scalar, vision tower — are read directly from the unquantized bfloat16 source. This is simpler and more reliable than decoding F32/Q6_K tensors from GGUF.

Negative Q4_0 Scales

Q4_0 uses d = max/8 which can be negative when all values in a group are negative (~49% of groups). The CT Marlin kernel handles negative bfloat16/float16 scales correctly.

Bugs Fixed

The following issues were discovered and fixed during development:

1. F32 tensor data corruption: GGUFReader decodes F32 in HF layout, but old code did a bytes→reshape→transpose round-trip that scrambled data. Fixed by reading non-Q4_0 tensors directly from the unquantized bf16 source instead of GGUF passthrough.

2. Q6_K dequantization errors: Similar reshape issue for embed_tokens + Q6_K quantization loss (483M values differ from true bf16, max error 0.007). Fixed by reading embed_tokens from the unquantized bf16 source.

3. Config actorder: "static": Triggered Marlin g_idx permutation without providing weight_g_idx, scrambling weights at inference. Removed from config.

4. dtype consistency: Scale dtype, passthrough tensor dtype, and config dtype all follow --dtype uniformly.

Tensor Counts