Qwen3.6 27B NVFP4 MTP GGUF
Native llama.cpp conversion of unsloth/Qwen3.6-27B-NVFP4. This is a separate artifact from the existing NVIDIA-source file in this repository.
Files
| File | Source | Format |
|---|---|---|
qwen3.6-27b-nvfp4.gguf | NVIDIA NVFP4 | Canonical NVFP4 download; bundled MTP preserved |
qwen3.6-27b-nvfp4_unsloth.gguf | Unsloth NVFP4 | Native NVFP4 FFNs, source FP8 tensors stored as Q8_0, bundled MTP preserved |
Size: 21.58 GiB. SHA256: DDCE600959ED99C16092EF24E8885AE18FD0D0B8A53EF92976BE1A395194FBD3.
The source checkpoint uses a mixed compressed-tensors layout. The conversion keeps packed NVFP4 FFN weights native and stores the source FP8 weights as Q8_0 to leave usable KV-cache headroom on an RTX 5090. The source mtp.* block is preserved for draft-mtp speculative decoding.
RTX 5090 Result
Windows 11, RTX 5090, llama.cpp b9851, ctx=200000, q4_0 target/draft KV, draft-mtp n=2, no-thinking, identical BookContext prompt fixture, and one measured 1024-token completion.
| Source | 10k decode tok/s | 200k decode tok/s | VRAM after | Temperature after |
|---|---|---|---|---|
| NVIDIA source GGUF | 69.0 | 42.2 | 30.8 GiB | 52 C / 61 C |
| Unsloth source GGUF | 72.8 | 44.1 | 26.1 GiB | 46 C / 60 C |
decode tok/s is llama.cpp slot print_timing generation throughput after prompt prefill, matching the repository's all-model chart. The Unsloth conversion was about 6% faster at 10k and 5% faster at 200k in this run, while finishing 6 C cooler at 10k and 1 C cooler at 200k.
Quick Start
Download and run the canonical NVIDIA-source model directly from Hugging Face:
llama-cli -hf neko-legends/Qwen3.6-27B-NVFP4-MTP-GGUF:NVFP4
Download and run the Unsloth-source model:
llama-cli -hf neko-legends/Qwen3.6-27B-NVFP4-MTP-GGUF:NVFP4_UNSLOTH
Equivalent local server configuration for the Unsloth-source file:
llama-server \
--model qwen3.6-27b-nvfp4_unsloth.gguf \
--alias qwen36-27b-unsloth-nvfp4-mtp-gguf \
--host 0.0.0.0 \
--port 39196 \
--device CUDA0 \
--gpu-layers all \
--gpu-layers-draft all \
--ctx-size 200000 \
--cache-type-k q4_0 \
--cache-type-v q4_0 \
--cache-type-k-draft q4_0 \
--cache-type-v-draft q4_0 \
--flash-attn on \
--parallel 1 \
--cont-batching \
--jinja \
--reasoning off \
--spec-type draft-mtp \
--spec-draft-n-max 2 \
--spec-draft-p-min 0.0
Companion Scripts
Windows download, conversion, Hermes setup, launcher, and benchmark scripts are in neko-legends/nvidia-local-llm-profiles. The repository also includes the temporary mixed-NVFP4 llama.cpp converter compatibility patch used for this release.
Apache 2.0, inherited from the Unsloth source release.