Update 2025-05-06: Replaced chat_template in tokenizer_config.json with the fixed version from froggeric/Qwen-Fixed-Chat-Templates.

Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-FP8

Vision-capable FP8-quantized abliterated Qwen3.6-35B-A3B (MoE, hybrid mamba/attention), built from the Huihui Claude-4.7-Opus-abliterated variant. Targets Nvidia DGX Spark and other FP8-capable hardware (~80 GB VRAM for full 262k context).

Model Lineage

• Base: Qwen/Qwen3.6-35B-A3B (BF16, hybrid linear-attention + full-attention MoE with 40 layers, 256 experts, vision encoder)
• Abliterated (refusals removed) by Huihui: huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated
• Claude-4.7-Opus-style tuning by Huihui: huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated
• This repo: FP8 quantization matching the format of Qwen/Qwen3.6-35B-A3B-FP8

Why FP8

Qwen3.6-35B-A3B in BF16 is ~72 GB on disk. FP8 cuts that to ~37 GB while preserving vision layers and precision-sensitive modules in BF16. The expected throughput uplift on DGX Spark is on par with what we saw for Qwen3.5 (31 → 51 t/s, ~65%).

Quantization Details

Scheme: native FP8 blockwise, identical on-disk format to the official Qwen/Qwen3.6-35B-A3B-FP8.

Quantized (weights → FP8 e4m3, per-block [128, 128] scales):

• All 2D Linear *.weight in language layers that aren't in the exclusion list, including:
  • self_attn.{q,k,v,o}_proj (full-attention layers)
  • linear_attn.{in_proj_qkv, in_proj_z, out_proj} (linear-attention / mamba layers)
  • mlp.shared_expert.{gate,up,down}_proj
  • All 256 experts per MoE layer, un-fused to match Qwen's official per-expert layout:
    • mlp.experts.{0..255}.{gate_proj, up_proj, down_proj}.weight

Kept in BF16 (matches Qwen's modules_to_not_convert):

Notable Implementation Notes

• Source experts were fused 3D (mlp.experts.gate_up_proj[256, 1024, 2048], mlp.experts.down_proj[256, 2048, 512]) — we un-fuse them to the per-expert layout the official Qwen FP8 uses (mlp.experts.{E}.{gate, up, down}_proj.weight). This is what vLLM's Fp8 MoE loader expects.
• Streaming quantization: processed one source shard at a time on the GPU; peak host memory ~6 GB. Avoids the llmcompressor pitfall where peak VM grew to 168 GB during the Compressing phase and got OOM-killed on the 128 GB DGX Spark unified-memory budget.
• Sanity check: round-trip dequantization median relative error ~2.16% per tensor across sampled MoE expert and attention layers — in line with the E4M3 blockwise profile observed on the sibling repo.
• Hybrid architecture preserved: this model has 10 full-attention layers (3, 7, 11, 15, 19, 23, 27, 31, 35, 39) interleaved with 30 linear-attention / mamba layers; both are quantized in the same way but only the full-attention layers have self_attn.{q,k,v,o}_proj.

Numbers

Loading

from transformers import AutoModelForImageTextToText, AutoProcessor
model = AutoModelForImageTextToText.from_pretrained(
    "batsclamp/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-FP8",
    dtype="auto",
    device_map="auto",
)
processor = AutoProcessor.from_pretrained("batsclamp/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-FP8")

For vLLM, point at the repo — the quantization_config is already correctly set (quant_method: fp8, weight-block [128, 128], dynamic activations).

Disclaimer

Abliterated model. Not recommended if you expect a polite corporate assistant.