[!NOTE] 1. Qwen3.6 35B A3B FP8 / Q4 K M / MTP / GGUF Qwen3.6 35B A3B FP8 transformers converted to .gguf Quantized to Q4 K M MTP recovered and working on latest llama.cpp main repo (natively) Why FP8 + MTP? You want near BF16 fidelity with ~50% memory savings and improved bandwidth efficiency. This model was exported from FP8 (F8 E4M3) precision provided by Qwen, then quantized to Q4 K M for GGUF compatibility. When running with llama.cpp, you get efficient inference via optimized integer kernels and reduced memory footprint. [!NOTE] 2. This model leverages the FP8 (F8 E4M3) export base to deliver: Near BF16 fidelity : Maintains numerical stability for most inference tasks while reducing precision overhead. ~50% memory savings : Q4 K M uses ~4 bits per weight vs BF16's 16 bits, enabling larger contexts or batch sizes on the same hardware. Improved bandwidth efficiency : Less data movement translates to higher throughput, especially in memory bound scenarios. [!NOTE] 3. Synergy with Multi Token Prediction (MTP) When combined with MTP, the reduced memory footprint unlocks additional efficiency gains: The lighter weight representation alleviates pressure on the KV cache , allowing MTP…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy