GLM-5.1-Abliterated-Dynamic-IQ3-340-GGUF

GGUF-quantized version of helixdouble/GLM-5.1-Abliterated (base: zai-org/GLM-5.1-FP8).

744B parameters MoE / 79 layers / 256 experts / 40B active / context length 202,752

⚠️ CRITICAL: Known Issue — Garbage Output on CUDA (As of 2026-05-18)

This model currently produces garbage output on all tested CUDA configurations. The build-time "smoke test" only measured token generation speed (tok/s) — output quality was never verified.

Symptom

• Greedy decoding (temperature=0): always emits token 0 (!) repeatedly: !!!!!!!!!!!!!!!!!!!!
• Sampling: random character garbage (e.g., H2%(@G&=6#9C4,')

Affected Configurations (all tested, all failed)

Root Cause

• ik_llama.cpp Issue #1045: Unresolved bug — GLM-family models produce garbage on CUDA. Bisected to fused expert selection commits (28d3e63).
• ik_llama.cpp Issue #1616: NaN logits on CUDA with GLM-5.1 when using Q4_0 KV cache.
• GLM-DSA lacks GGML_PREC_F32 workarounds present in GLM4/GLM4_MOE.
• Blackwell (SM120) architecture may exacerbate the issue.

Additional Findings

• Q8_0 base model (747GB) crashes with segmentation fault on CPU-only inference (-ngl 0).
• mainline llama.cpp cannot load this model — ik_llama.cpp uses extended ggml type 141 (IQ3_KS) not supported upstream.
• Performance is excellent (~51 tok/s prompt eval, ~40 tok/s generation) — the model runs fast, just produces wrong tokens.

🔧 Call for Community Help

If you have successfully run GLM-5.1 (or GLM-4.5-Air) with IQ quantization on ik_llama.cpp CUDA, please share:

1. Exact commit hash and build flags
2. Launch command line
3. GPU architecture (compute capability)
4. Any source patches applied

Please comment on ik_llama.cpp Issue #1045 or open a PR to this repo.

Recommended Next Steps

1. Wait for ik_llama.cpp fix — Track Issue #1045.
2. Try bati.cpp v0.2.0+ — Official GLM-5.1 support is planned.
3. Consider Q4_K_M or Q5_K quantization — Non-IQ types may be less affected (unconfirmed).
4. Consider alternative models — DeepSeek V4, Qwen 3.6, etc.

Quantization Details

Custom Quantization Strategy

High-importance tensors kept at high precision; routed experts heavily compressed:

Inference

⚠️ The following commands are provided for reference, but output quality is currently broken on all tested configurations. See "Known Issue" above.

Requirements

• RAM: 512GB+ recommended
• GPU VRAM: 4-GPU config recommended (96GB+ each)
• Supported engines: ik_llama.cpp or bati.cpp
• Storage: 350GB+ free for GGUF files

Inference Commands

1. Fast Config (Recommended when fixed)

Full GPU offload. Optimized for speed, stable up to ctx=32768.

CUDA_VISIBLE_DEVICES=0,1,2,3 ./build/bin/llama-server \
  -m /path/to/GLM-5.1-Abliterated-Dynamic-IQ3-340-00001-of-00018.gguf \
  --host 127.0.0.1 --port 8081 \
  --ctx-size 32768 \
  -ngl 999 -sm layer --device CUDA0,CUDA1,CUDA2,CUDA3 \
  -ctk q8_0 -ctv q8_0 \
  -t 24

2. Fallback Config — Long Context

CPU offload mix. Achieves ctx=65535. Use when long context is mandatory.

CUDA_VISIBLE_DEVICES=0,1,2,3 ./build/bin/llama-server \
  -m /path/to/GLM-5.1-Abliterated-Dynamic-IQ3-340-00001-of-00018.gguf \
  --host 127.0.0.1 --port 8081 \
  --ctx-size 65535 \
  -ngl 70 -sm layer --device CUDA0,CUDA1,CUDA2,CUDA3 \
  -ctk q4_0 -ctv q4_0 \
  -t 24

llama-cli (Chat Mode)

CUDA_VISIBLE_DEVICES=0,1,2,3 ./build/bin/llama-cli \
  -m /path/to/GLM-5.1-Abliterated-Dynamic-IQ3-340-00001-of-00018.gguf \
  --ctx-size 32768 \
  -ngl 999 -sm layer --device CUDA0,CUDA1,CUDA2,CUDA3 \
  -ctk q8_0 -ctv q8_0 \
  -t 24 -cnv

Benchmark (4× RTX PRO 6000 Blackwell 96GB, verified 2026-05-17)

• Fast config (-ngl 999): ctx=32768 is the limit. Beyond this, CUDA0 compute buffer OOM.
• Fallback config (-ngl 70): Offloads last 9 layers to CPU to achieve ctx=65535. Speed drops ~47%.
• Flash Attention is enabled by default in ik_llama.cpp (no explicit flag needed).

Build Report

See BUILD_REPORT.md for detailed build pipeline, environment, and the full list of tested configurations.

Credits

This model is based on:

• Base model: zai-org/GLM-5.1-FP8 — Z.ai's 754B MoE model
• Abliteration: helixdouble/GLM-5.1-Abliterated — Precision abliteration via 250 Optuna trials + healing LoRA
• Abliteration reproduction dataset: helixdouble/glm-5.1-abliteration-trials-250
• GGUF conversion & quantization tools:
  • bati.cpp — GLM-DSA compatible converter
  • ik_llama.cpp — Dynamic IQ3 quantization engine
• Calibration data: heliXdouble bartowski_calibration_v5 + Lna-Lab additions

SHA256

# Generate with:
sha256sum *.gguf > SHA256SUMS.txt

License

AGPL-3.0 (following the base model and abliteration artifacts)