s-batman/Ornith-1.0-9B-NVFP4-MTP-GGUF

Three honest quantizations of deepreinforce-ai/Ornith-1.0-9B with Multi-Token Prediction (MTP) heads grafted from Qwen3.5-9B-MTP, packaged as three GGUF variants for llama.cpp.

Designed for NVIDIA Blackwell GPUs (sm_120 / sm_121) including the RTX PRO 6000 and the DGX Spark (GB10). NVFP4 and MXFP4 are dequantized natively by Blackwell tensor cores; the grafted MTP heads enable draft-mtp speculative decoding for significant decode throughput uplift.

Naming note (read first): the repo is named -NVFP4-MTP-GGUF because that was the original (incorrect) name. The repo now hosts three quant files of three different formats — see the Provided Files table. Pick the file whose name matches its actual tensor layout.

Original Model

Ornith-1.0-9B is a self-improving agentic-coding model released by the DeepReinforce team, post-trained via RL on top of Qwen3.5-9B. It is the dense sibling of Ornith-1.0-35B (which is MoE). Both share the Qwen3.5 hybrid-attention trunk (linear SSM + full-attention every 4th layer) and the same reasoning_content block-before-answer output format.

• Architecture: Qwen3.5 dense (qwen3_5_text), 32 trunk layers + 1 embedded MTP layer (counted as blk.32), hidden_size 4096
• Parameters: 9.2B total / ~9.2B active (dense — all params active per token)
• Vocabulary: 248,320 tokens (multimodal vocab preserved; vision tower not included in this GGUF)
• License: MIT (inherited from upstream)
• Citation: see Citation below

What's in this repo

This repository hosts three honestly-labeled GGUF quants — all built from the same BF16 trunk with the same MTP graft, then quantized with three different llama-quantize ftypes:

MTP block in all three files comes byte-for-byte from a NVFP4-quantized source, so 8 of the 15 MTP tensors (the linear attn/ffn weights in blk.32) are NVFP4 even in the MXFP4 and Q8_0 trunk files. This is intentional — re-quantizing the MTP block with a different ftype was not done to preserve the graft's proven correctness.

What is NVFP4 vs MXFP4?

Both are 4-bit block-floating-point formats with E2M1 element data, but they differ in scale factor layout and total bits:

• MXFP4 (OCP standard, GGML type 39): 32 elements per block, E8M0 scale (1 byte per 32 elements). 4.25 bpw for the quantized data, ~4.50 bpw total with header overhead.
• NVFP4 (NVIDIA proprietary, GGML type 40): 16 elements per sub-block × 4 sub-blocks per block, E4M3 scales (4 bytes per 64-element block). 4.50 bpw for the quantized data, ~4.75 bpw total.

NVFP4 is the Blackwell-native format (hardware-dequantized directly by Blackwell tensor cores); MXFP4 is the industry-standard OCP spec. On Blackwell hardware, NVFP4 is slightly faster due to the tensor-core-native path; off-Blackwell, both formats require software dequantization and perform similarly.

The previous version of this repo (commit history before 2026-06-29) had a single file labelled ornith-1.0-9b-NVFP4-MTP.gguf that was actually all Q8_0 + F32 internally (zero FP4 tensors). That naming was wrong — the file has been re-quantized as the three honest variants above.

Provided Files

The video_preprocessor_config.json file is also present but is informational only.

Quantization Details

Quantization Tooling

All three quants were produced with a patched llama-quantize from llama.cpp build d2462f8f7 (commit hash 9590 upstream). The patch:

1. Registers the LLAMA_FTYPE_MOSTLY_NVFP4 = 39 ftype in the tools/quantize/quantize.cpp QUANT_OPTIONS table (upstream only registers MXFP4_MOE = 38).
2. Routes all non-output / non-token-embedding tensors to the chosen 4-bit ftype for dense models, including 2D linears. Upstream's MXFP4_MOE and NVFP4 heuristics only target 3D MoE expert tensors (blk.*.ffn_*_exps.weight with ne[2] > 1), so on a dense model like Ornith-9B the upstream code falls back to Q8_0 for everything — leaving the file with file_type=39 but zero actual FP4 tensors. The patched version makes the file_type label actually match the contents.

The patch is in /opt/llama/llama-cpp-cuda-build/src/llama-quant.cpp at lines ~463–478. The build lives at /opt/llama/llama-cpp-cuda-build/build-cuda128-sm120/bin/llama-quantize.

What is MTP and why graft it?

Multi-Token Prediction uses a small auxiliary head to predict multiple tokens ahead. At inference, those predictions become a draft that the main model verifies in a single forward pass — speculative decoding with zero quality loss (output distribution is identical to non-MTP) and significant decode speedup when drafts are accepted.

Ornith-1.0-9B is a Qwen3.5-9B post-trained variant. Its trunk shares the same parameter shapes and base tensor layout as the MTP-enabled Qwen3.5-9B, including the same qwen3_5_text MTP block position (15 tensors at blk.32.*, ~125 MB total). Because the MTP head operates on the same hidden state and embedding space as the trunk, the grafted heads work without any additional training — the MTP drafts the next token given the trunk hidden state, the trunk then verifies. No re-tuning required.

Performance

(Performance numbers are reported for the NVFP4 variant, which is the production config. Q8_0 will be ~2× slower due to memory bandwidth; MXFP4 should be within a few percent of NVFP4 on Blackwell.)

RTX PRO 6000 (Blackwell, sm_120, 97 GB GDDR7, ~1.79 TB/s)

Single-slot, 8k context, temp=0.9, top_p=0.95, top_k=20, min_p=0.01, KV cache q8_0, b=ub=2048:

The 9B at c=8k with 5 slots (c=524288, parallel=5) fits comfortably in the 6000 Pro 97 GB VRAM (~5 GB model + ~10 GB KV/q8_0 + ~5 GB MTP working set), leaving headroom for long-context workloads.

DGX Spark (Blackwell GB10, sm_121, 121 GB unified LPDDR5X, ~273 GB/s)

The DGX Spark unified memory and slower bandwidth make it ideal for memory-bandwidth-bound dense models. Expect ~150–170 tok/s decode at single-slot 8k context with MTP enabled (proportional to the bandwidth ratio vs the 6000 Pro).

Usage with llama.cpp

Requirements

• llama.cpp build with NVFP4/MXFP4 support, sm_120 / sm_121 CUDA (Blackwell)
• CUDA toolkit with Blackwell support
• Build with -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="120;121" (set to your GPU compute capability)

Server (recommended for production)

RTX PRO 6000 / single-GPU Blackwell, 5 slots, 512k context, NVFP4 variant:

llama-server \
  -m ornith-1.0-9b-NVFP4-MTP.gguf \
  --host 0.0.0.0 --port 8080 --slots --metrics \
  -t 64 -cb --no-warmup --no-mmap --mlock \
  --jinja -fa on -ctk q8_0 -ctv q8_0 \
  -b 2048 -ub 2048 -c 524288 -np 5 -ngl 99 \
  --chat-template-file chat_template.jinja \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --temp 0.9 --top-p 0.95 --top-k 20 --min-p 0.01 --repeat-penalty 1.1

For Q8_0 or MXFP4, just swap the -m filename.

CLI

llama-cli \
  -m ornith-1.0-9b-NVFP4-MTP.gguf \
  --chat-template-file chat_template.jinja \
  -p "What is 17 times 23?" \
  -ngl 99 \
  --temp 0.9 --top-p 0.95 --top-k 20 --min-p 0.01 \
  --spec-type draft-mtp --spec-draft-n-max 3

Direct download with llama.cpp

llama-cli \
  --hf-repo s-batman/Ornith-1.0-9B-NVFP4-MTP-GGUF \
  --hf-file ornith-1.0-9b-NVFP4-MTP.gguf \
  -p "What is 17 times 23?"

Important Notes

• Blackwell strongly recommended for the 4-bit variants. NVFP4 is hardware-native on Blackwell; MXFP4 is hardware-native on Blackwell via the OCP MX path. On other GPUs (Ampere/Hopper/Ada/AMD/Intel/CPU), use the Q8_0 variant — software-dequantized 4-bit is slower than Q8_0 on those architectures.
• --no-mmap recommended on unified-memory machines (DGX Spark) to avoid mmap-related slowdowns.
• KV cache type q8_0 is the production setting. F16 KV uses ~2× more memory for negligible quality change on long context.
• 9B vs 35B: if your workload is light (single-turn chat, short context) the 35B will give you higher quality. For high-throughput multi-slot workloads (5+ concurrent users, agentic loops) the 9B NVFP4+MTP is a better fit — lower VRAM, higher tokens-per-second per slot, and the MTP speedup is more pronounced on the dense model.

How the MTP graft was made

The MTP heads were transferred byte-for-byte from FreedomAISVR/Qwen3.5-9B-MTP-NVFP4-GGUF (which mirrors Unsloth's MTP-enabled Qwen3.5-9B GGUF, but in NVFP4 form) into the deepreinforce-ai Ornith-1.0-9B BF16 GGUF before quantization.

1. Source GGUF (FreedomAISVR MTP-enabled Qwen3.5-9B NVFP4) has 442 tensors and qwen35.block_count=33. The MTP layer sits at blk.32 as a full 15-tensor block:

   • 11 trunk-style tensors at blk.32.* (attn_q/k/v/output, ffn_gate/up/down, plus 3 norms)
   • 4 MTP-specific tensors at blk.32.nextn.*: eh_proj.weight, shared_head_norm.weight, enorm.weight, hnorm.weight

   The MTP block is embedded at blk.32 (the trunk's last layer slot) — not appended as a virtual layer. qwen35.block_count=33 means 32 trunk layers + 1 fused MTP layer. Note: this is in contrast to the 35B MoE case, where the MTP block is appended after blk.40 as an independent set of 20 tensors.

2. Destination GGUF (Ornith-1.0-9B BF16) has 427 tensors, qwen35.block_count=32, no MTP tensors. Its last tensor is output_norm.weight.

3. Graft procedure (per-tensor copy):

   • Parse source GGUF: discover all blk.32.* tensors (15 total), record each (name, dims, type, data-region-relative offset, byte_count).
   • Parse destination GGUF: get data-region start, data size, tensor count.
   • Build new KV block: copy dst KV verbatim except bump qwen35.block_count 32→33 and append qwen35.nextn_predict_layers=1 (U32). The KV key prefix (qwen35.* here, vs qwen35moe.* in the 35B) is discovered dynamically by suffix matching.
   • Build new tensor index: 427 dst entries (offsets preserved, since dst tensor index is data-region-relative) plus 15 new blk.32.* entries with sequential 32-byte-aligned offsets starting after the dst data block.
   • Write output: header (24 B) | new KV | new tensor index | padding to 32 B align | dst data block (verbatim) | 15 blk.32 tensors.

   Critical offset gotcha: GGUF tensor offsets are data-region-relative, not file-absolute. To read a tensor payload you MUST seek to (data_start + offset). A bare seek(offset) lands near byte 0 and silently copies trunk tensors — producing a GGUF that loads but has garbage in the MTP slot.

4. Quantize the merged BF16 GGUF with the patched llama-quantize for each of the three target ftypes.

The graft script is at /home/steven/Research/ornith9b-build/graft_mtp_v2.py.

Licensing

This model is licensed under MIT, the same license as the original deepreinforce-ai/Ornith-1.0-9B. See LICENSE for the full text.

The MTP heads originate from Qwen/Qwen3.5-9B (Apache 2.0). The MTP block was sourced via the FreedomAISVR/Qwen3.5-9B-MTP-NVFP4-GGUF redistribution, which is permitted under the original Apache 2.0 terms.

Citation

@misc{ornith-9b,
    title = {{Ornith-1.0-9B}: Agentic Coding, Open to All},
    url = {https://deep-reinforce.com/ornith_1_0.html},
    author = {{DeepReinforce Team}},
    year = {2026}
}

@misc{qwen3.5-9b,
    title = {{Qwen3.5-9B}: Hybrid-Attention Dense Language Model},
    url = {https://huggingface.co/Qwen/Qwen3.5-9B},
    author = {{Qwen Team}},
    year = {2026}
}

Acknowledgements

• DeepReinforce Team for training and releasing Ornith-1.0-9B
• Alibaba Qwen Team for the Qwen3.5 base models
• FreedomAISVR for the MTP-enabled NVFP4 GGUF (mirror of Unsloth's MTP redistribution) that made the graft possible
• llama.cpp for NVFP4/MXFP4 support and the draft-mtp speculative decoding implementation