Qwen3.5-122B-A10B-NVFP4-MTP-GGUF

NVFP4 GGUF of Qwen/Qwen3.5-122B-A10B with the MTP (Multi-Token Prediction) head retained. Structurally identical to the sibling artifact Incarnas/Qwen3.5-122B-A10B-NVFP4-GGUF plus one additional layer block carrying the MTP draft weights, enabling self-speculative decoding via llama.cpp's --spec-type draft-mtp path.

122B total parameters, ~10B active per token via 256 MoE experts (8 routed plus 1 shared). 48 base transformer blocks plus 1 MTP block (block index 48). 77 GiB main GGUF, 871 MiB mmproj sidecar carrying the vision tower.

For reproducibility / audit / debugging details — full bug-discovery narrative, per-test bench tables, hardware specs, failure modes encountered during this run, and step-by-step reproduction recipe — see the companion TECHNICAL_REPORT.md (or the PDF version on the methodology repo).

The full methodology toolchain — the converter patch, bench scripts, raw bench JSONs, power telemetry CSVs, and gsm8k samples — lives at bit-incarnas/nvfp4-mtp-conversions (v1.0).

Updates

• 2026-05-18 — The converter patch required by this release has been merged upstream as ggml-org/llama.cpp#23237 (master commit 1867a0c69, first contained in release tag b9208). The artifact itself is unchanged; reproducers building against b9208+ no longer need the local patch step. The patch remains documented in the methodology repo for anyone reproducing against the original pre-merge b9187 build.
• 2026-05-16 (later) — Added a public cross-link from the Bench reproduction section to the benchmark_results/ tree on the methodology repo, where copies of the bench JSONs + power CSVs from this release's bench window are committed.
• 2026-05-16 — initial release. Conversion target: llama.cpp b9187 (post-PR-22673 merge), which added Qwen3.5-MoE MTP tensor remap in convert_hf_to_gguf.py. Bench: AR decode 79.8 t/s, MTP n=2 decode 116.5 t/s (+46%), spec acceptance 2.46 eff tok/decode.

What this artifact adds vs. the no-MTP sibling

The standard Incarnas/Qwen3.5-122B-A10B-NVFP4-GGUF GGUF is built from the same NVFP4 source weights but with MTP tensors stripped (the converter version at the time of that release did not yet handle the mtp.* prefix for the qwen3_5_moe arch). PR #22673 fixed the converter at merge.

This artifact retains the MTP block at layer 48. Concretely the additions are:

Together these add ~6 GiB on disk vs the no-MTP sibling. GGUF metadata: qwen35moe.block_count = 49, qwen35moe.nextn_predict_layers = 1.

Source and conversion

• Source weights: txn545/Qwen3.5-122B-A10B-NVFP4 (NVIDIA ModelOpt NVFP4 safetensors, ~74 GiB, 149,765 tensors including 785 under the mtp.* prefix).

• Converter: llama.cpp convert_hf_to_gguf.py at commit 0253fb21f (build b9187), the first build with PR #22673 (MTP support) on master.

• A one-line local patch to conversion/qwen.py (committed at patches/qwen35_mtp_bid_fix.patch in the methodology repo) was required on top of mainline. _Qwen35MtpMixin.modify_tensors() (around line 597 at commit 0253fb21f) remaps the tensor NAME from mtp.layers.{bid} to model.layers.{bid + n_layer} but doesn't update the bid argument before yielding to super. The Qwen2MoE expert-stacker (Qwen2MoeModel.modify_tensors, around line 87) uses bid as the key into its expert cache and rebuilds expected tensor names from bid directly, so the renamed-but-bid-unchanged tensor gets cached at the wrong slot and the stacker tries to look up layer-0 keys in a cache populated with layer-48 keys, crashing with KeyError: 'model.layers.0.mlp.experts.0.down_proj.weight'. The fix adds one line:

  --- a/conversion/qwen.py
  +++ b/conversion/qwen.py
  @@ class _Qwen35MtpMixin:
       def modify_tensors(self, data_torch, name, bid):
           if name.startswith("mtp."):
               n_layer = self.hparams["num_hidden_layers"]
               if name.find("layers.") != -1:
                   assert bid is not None
                   name = name.replace(f"mtp.layers.{bid}", f"model.layers.{bid + n_layer}")
  +                bid = bid + n_layer
               else:
                   ...
  

  The bug doesn't manifest for text-only Qwen3.5-MoE sources (e.g., the ones Unsloth converted for their MTP releases) because their safetensors use the bare model.layers.{N}.* naming and the MoE expert-stacker happens to consume the layer-0 cache before the MTP path runs. It does manifest for Qwen3_5MoeForConditionalGeneration (multimodal) sources where tensors arrive under model.language_model.layers.{N}.* and the strip-prefix path changes the order in which the cache fills. Merged upstream as ggml-org/llama.cpp#23237 on 2026-05-18 (master commit 1867a0c69, first release tag b9208). Reproducers building from llama.cpp b9208+ no longer need to apply the patch -- the fix ships in mainline. The patch in patches/qwen35_mtp_bid_fix.patch remains available for anyone reproducing against the original pre-merge b9187 build that this v1.0 GGUF was produced from.

What this conversion includes

llama.cpp converter changes that apply to this artifact:

• PR #22673 (May 2026, merged): Qwen3.5-MoE MTP tensor remap in the converter and --spec-type draft-mtp runtime support.
• PR #22611 (May 2026): Q/K RoPE permutation in the NVFP4 repack path.
• PR #22196 (Apr 2026): Blackwell-native NVFP4 runtime path eligibility. Set BLACKWELL_NATIVE_FP4=1 at runtime on a supported GPU.
• PR #22247 (Apr 2026): mixed-precision ModelOpt handling for the FP8 exclude list.
• PR #22356 (Apr 2026): removed unused input_scale from FP8-dequanted modelopt tensors.

Vision tower preserved as a separate mmproj GGUF, identical to the no-MTP sibling's mmproj (vision tower itself is unchanged between releases). Chat template embedded. Thinking mode is enabled by default per Qwen's chat template.

Mixed-precision layout

ModelOpt's hf_quant_config.json excludes the following modules from NVFP4. They stay at the source dtype (BF16 with FP8 scales on the quantized portions):

• lm_head (output projection).
• All linear-attention modules (the Gated DeltaNet layers, which carry SSM-style state).
• All mlp.shared_expert_gate modules (MoE router gates).
• Self-attention on the full-attention layers.
• MTP-block weights (the new block 48) follow the same exclude pattern: BF16/FP8 on the same set of modules within the MTP block.

KV cache: kv_cache_quant_algo: FP8 per the source config. NVFP4 group size: 16 elements. NVFP4 scale dtype: FP8 (E4M3).

The "two NVFP4 formats" gotcha

If you try to convert your own NVFP4 source and hit:

NotImplementedError: Quant format 'nvfp4-pack-quantized' for method 'compressed-tensors' is not yet supported

the cause is that two different toolchains both produce things called "NVFP4" and only one is supported by llama.cpp today.

Check quant_method in the source config.json before downloading. ModelOpt sources work today.

File contents

How to run

MTP self-speculative decoding (single-user, max decode rate):

llama-server \
  -m Qwen3.5-122B-A10B-NVFP4-MTP.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  -ngl 999 \
  -c 32768 \
  -fa on \
  -ctk q8_0 -ctv q8_0 \
  --port 8080

Vision (no MTP, multimodal input enabled):

llama-server \
  -m Qwen3.5-122B-A10B-NVFP4-MTP.gguf \
  --mmproj mmproj-Qwen3.5-122B-A10B-NVFP4.gguf \
  -ngl 999 \
  -c 32768 \
  --port 8080

Docker:

docker run --rm -it --gpus all \
  -v $PWD:/models \
  -p 8080:8080 \
  -e BLACKWELL_NATIVE_FP4=1 \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m /models/Qwen3.5-122B-A10B-NVFP4-MTP.gguf \
  --spec-type draft-mtp --spec-draft-n-max 2 \
  -ngl 999 -c 32768 -fa on

Toggle thinking mode via enable_thinking in chat params if your client supports it.

Constraints and gotchas

These are PR #22673 limitations as of the merge commit:

• --mmproj is incompatible with --spec-type draft-mtp. The loader hangs during init. Pick one of the two modes per server instance.
• n_parallel must be 1 under --spec-type draft-mtp. Multi-slot serving disables MTP.
• cache_reuse is auto-disabled by the server when --spec-type draft-mtp is active.
• --spec-draft-n-max sweet spot is 2 on this artifact at the measured envelope. n=3 and n=4 are within 1% of n=2 on decode rate but use more VRAM for the draft buffer and slightly more compute per accepted token.

Performance

Measured 2026-05-16 on Blackwell Pro 96 GiB (RTX PRO 6000 Blackwell Max-Q, 300 W TDP) at llama.cpp commit 0253fb21f (build b9187), 256k context, KV cache q8_0/q8_0, --flash-attn on, BLACKWELL_NATIVE_FP4=1. Decode and prefill numbers are medians from tools/bench_decode_prefill.sh with 3 reps per metric. AR mode: --spec-type none. MTP mode: --spec-type draft-mtp --spec-draft-n-max 2. Power and clock telemetry sampled at 500 ms intervals during each bench run.

Decode

The long-decode AR baseline of 79.77 t/s matches the no-MTP sibling's published 79.8 t/s within noise — the conversion is structurally additive, non-MTP path performance is preserved.

MTP wins most on the thinking-on regime (+46%) because the long structured reasoning trace is highly predictable: the draft head accepts many tokens per verifier step. Plain decode also gains (+19%) but less, because non-thinking output is more locally entropic. Short decode shows the largest absolute speedup (+57%) because draft-stage warmup costs are amortized over fewer total tokens.

Speculative-decode acceptance (MTP n=2)

Effective tokens per decode = Δtokens_predicted / Δn_decode from /metrics. AR would yield ~1.0; uniform 2-token speculation with 100% acceptance would yield 3.0 (1 verified + 2 accepted drafts). 2.46 corresponds to roughly 73% draft acceptance on this workload at --spec-draft-n-max 2. Acceptance is regime-dependent; thinking-on reasoning traces accept more than terse instruction-following.

Prefill and cache reuse

MTP carries a small per-step overhead during prefill — the drafter graph runs alongside the verifier graph — so the long-prefill rate is ~18% lower under draft-mtp. Decode amortizes this many times over: for any output longer than ~30 tokens the MTP path is net faster wall-clock. For pure short-prompt-short-output workloads (e.g. classifier-style chains) AR is the better pick.

Prefill curve at long context

Carried over from the no-MTP sibling (the prefill compute path is structurally unchanged — MTP affects per-step compute during decode, not during the prefill pass):

Clean monotonic decay. Per-token prefill compute grows with KV state.

VRAM at 256k context, q8 KV, no mmproj

The +5.7 GiB delta breaks down approximately as: ~4 GiB MTP-block weights (resident on top of the no-MTP sibling's 78.7 GiB) and ~1.7 GiB draft-state buffers (acceptance KV slice + drafter compute scratch, scales with context length and --spec-draft-n-max). Headroom at 256k is tight under MTP — drop to 131k context for a 4-5 GiB cushion if you also need to co-load adapters or run with --cpu-moe-draft off.

VRAM at 256k with mmproj loaded was not measured on this artifact under MTP because --mmproj and --spec-type draft-mtp are mutually exclusive at server startup. In AR mode with mmproj loaded, expect ~82.6 GiB resident (vision tower adds 248 MiB; image processing adds ~52 MiB peak per 512x512 image).

Power, clocks, efficiency

GPU telemetry sampled at 500 ms intervals during long-decode-thinking-on:

MTP delivers +51% tok/J on long-decode-thinking. Same wall power, more tokens per joule because the GPU spends less time in compute-saturated regime per accepted token. Lower utilization, lower temperature, lower clock under MTP also reflect that compute is no longer the binding constraint when draft acceptance amortizes it across multiple tokens per verifier step.

BLACKWELL_NATIVE_FP4=1 was set in both runs. MoE decode here is memory-bandwidth-bound rather than compute-bound, so the FP4 native tensor-core path does not materially change decode rate on this rig — the flag is still recommended on Blackwell because differences may show on workloads dominated by prefill compute (which this measurement covers but at a single prompt size).

Hardware and environment

• GPU: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (96 GiB GDDR7, 300 W TDP, PCIe 5.0 x16)
• CPU: AMD Ryzen Threadripper 7960X (24 cores / 48 threads, SMT enabled)
• RAM: 128 GiB DDR5 ECC RDIMM @ 6400 MHz
• Storage (model file at bench time): NFS over 10 GbE direct from TrueNAS, mounted at /media/temple
• OS / driver: Linux 7.0.0-15-generic, NVIDIA driver supporting BLACKWELL_NATIVE_FP4
• llama.cpp image: locally-built from commit 0253fb21f (build b9187), .devops/cuda.Dockerfile --target server

Bench reproduction

The bench scripts used here are simple bash/python over urllib.request against the llama-server HTTP endpoint, no heavyweight harness dependencies. Three reps per metric, GPU telemetry sampled at 500 ms intervals via nvidia-smi.

Equivalent stand-alone invocations after starting the server (llama-server -m <gguf> --spec-type {none|draft-mtp} --spec-draft-n-max 2 -ngl 999 -c 262144 -fa on -ctk q8_0 -ctv q8_0 --port 8080):

# Decode + prefill + power (one run per mode)
tools/bench_decode_prefill.sh http://127.0.0.1:8080 v4-nvfp4-{ar,mtp-n2}

# Speculative-decode acceptance (wraps the above + reads /metrics deltas)
tools/spec_accept.sh http://127.0.0.1:8080 v4-nvfp4-mtp-n2

# gsm8k chat-mode probe (uses /v1/chat/completions with enable_thinking)
tools/bench_chat_math.py http://127.0.0.1:8080 v4-nvfp4-{ar,mtp-n2} --limit 5

Output JSON per run lands at reports/bench/<label>.json and reports/chat-math/<label>.{json,md}. The decode-prefill bench writes a sibling <label>.power.csv with the 500 ms telemetry sweep.

Public copies of the JSONs + power CSVs from this release's bench window are committed at benchmark_results/ in the methodology repo.

Capability

MTP is verifier-driven: each speculative token is accepted only when the base model's logits agree at that position. Output is therefore statistically equivalent between MTP-on and AR (no-spec) modes at the same sampling settings. Capability scores measured on the no-MTP sibling carry over directly to this artifact for chat-completions thinking-off, which is the regime where MTP changes wall-clock but not token identity.

Empirical equivalence spot-check (this artifact)

A 5-sample gsm8k slice was run on this artifact under both --spec-type draft-mtp --spec-draft-n-max 2 and --spec-type none AR, using /v1/chat/completions with thinking enabled and bench_chat_math.py defaults (non-zero temperature, top-p sampling):

Four of five samples produced identical answers between MTP and AR. The diverging sample (Q4) reflects sampling-temperature variance at the per-token level rather than an MTP-induced quality regression; each mode samples its own trace through the same logit distribution at non-zero temperature. Sample size is too small to draw absolute-accuracy conclusions; the comparison establishes the equivalence claim, not the capability number.

Carried over from the no-MTP sibling

Full-volume capability runs from Incarnas/Qwen3.5-122B-A10B-NVFP4-GGUF (same NVFP4 source weights, same architecture, structural delta is one additional block; the verifier-driven equivalence argument applies):

The chat-vs-raw gap on ifeval (90% vs 44%, +46pp) is a methodology artifact, not a model property. Raw mode bypasses the chat template's role framing, so the model never engages instruction-following machinery for that task type. The chat-mode column is the right reference for chat-client and agentic-harness use cases; the raw column is the right reference for HF-leaderboard comparability. See Methodology notes below.

--spec-draft-n-max sweet-spot rationale

The choice of n=2 as the production setting is supported by sweeps on related artifacts at the same b9187 build:

Both peak at n=2 with a 1-3% plateau through n=4. This NVFP4-MTP artifact was measured at n=2 only; the broader sweep was not repeated because the cross-artifact pattern is established. Long-decode-thinking is the regime where n=2 wins by the largest margin; pure short-output regimes may benefit from n=3 or n=4 at the cost of more VRAM.

Not measured on this artifact

• Full chat-mode capability sweep under --spec-type draft-mtp (statistical equivalence holds in principle and on the 5-sample spot-check; a 100-sample sweep is a planned follow-up).
• --spec-draft-n-max sweep beyond n=2 (peak location inherited from cross-artifact data above).
• True cold-cache load from local NVMe.
• Aider polyglot under MTP (litellm model-registry workaround pending, same blocker as no-MTP sibling).
• Broader vision rubric against natural images.
• NoLiMa associative-retrieval at long context.

Measurement provenance

Every datapoint above falls into one of three categories:

Raw bench JSONs from this run live in the publish-pipeline working tree (see TECHNICAL_REPORT.md §8 for absolute paths), not in this HF repo. The companion TECHNICAL_REPORT.md, linked at the top of this card, carries the full bug-discovery narrative, per-test bench tables, hardware specs, failure modes encountered during this run, and the step-by-step reproduction recipe.

Methodology notes

This release ships under the bit-incarnas/nvfp4-mtp-conversions methodology repo (v1.0), which carries the full technical report (markdown | PDF), the local converter patch, raw bench JSONs + power telemetry, and the reproduction recipe.

The chat-vs-raw methodology gap and its implications for ifeval / instruction-following tasks are the same as on the no-MTP sibling. See Incarnas/Qwen3.5-122B-A10B-NVFP4-GGUF#methodology-notes for the full breakdown, raw-mode tables, and the bit-incarnas/chat-vs-raw-methodology reproduction recipe.

llama.cpp version requirements

This GGUF was produced by llama.cpp commit 0253fb21f (build b9187). It requires a llama.cpp build with PR #22673 merged (Qwen3.5-MoE MTP runtime support). Builds older than 2026-05-16 will load the model but cannot activate --spec-type draft-mtp; in that case the file falls back to behaving exactly like the no-MTP sibling and the MTP-block weights go unused.

For reproducing the conversion (not for running this GGUF): the v1.0 conversion required an additional one-line converter patch on top of the b9187 build. As of 2026-05-18 that patch has been merged upstream as PR #23237 in master commit 1867a0c69; the first release tag containing the fix is b9208. Reproducers using b9208+ can skip the local patch step entirely.

Attribution

• Base model: Qwen team, apache-2.0
• NVFP4 source weights: txn545/Qwen3.5-122B-A10B-NVFP4
• Quantization toolchain: NVIDIA ModelOpt
• GGUF conversion: llama.cpp convert_hf_to_gguf.py (with a one-line local fix to _Qwen35MtpMixin.modify_tensors() for multimodal-prefix sources)
• Sibling repo (no MTP): Incarnas/Qwen3.5-122B-A10B-NVFP4-GGUF