Nvidia-Qwen3.6-27B-NVFP4 - GGUF

Quantized GGUF versions of nvidia/Qwen3.6-27B-NVFP4. These were generated using llama.cpp (b9859).

• Nvidia-Qwen3.6-27B-NVFP4-A.gguf: The FFN and attention layers are NVFP4 quantized. The MTP draft layer is preserved at BF16. This was generated from BF16-Attn using a modified llama-quantize to re-quantize the BF16-upcast attention layers to NVFP4.
• Nvidia-Qwen3.6-27B-NVFP4-Q8.gguf: The NVFP4 FFN layers are preserved, while the FP8 attention layers are converted to Q8. This was generated using convert_hf_to_gguf.py.
• Nvidia-Qwen3.6-27B-NVFP4-BF16-Attn.gguf: The NVFP4 FFN layers are preserved, while the FP8 attention layers are upcast to BF16. This is the default conversion for BF16 because GGUF files do not support FP8. This was generated using convert_hf_to_gguf.py.

Quantizations provided

Perplexity test

I tested perplexity using llama-perplexity and Salesforce's wikitext-2-raw-v1.

Evaluation

The following models were evaluated for a fair comparison of capability, size and speed.

1: unsloth/Qwen3.6-27B-NVFP4 does not provide a GGUF. I used llama.cpp's conversion which passes through Unsloth's NVFP4 tensors.

These evaluations were run using lm_eval. The models were run in instruct (non-thinking) mode with the following parameters in llama-server (b9775):

ctx-size = 32768
cache-type-k = q8_0
cache-type-v = q8_0
top-p = 0.8
top-k = 20
min-p = 0
presence-penalty = 1.5
spec_type = draft-mtp
spec_draft_n_max = 2
chat-template-kwargs = {"enable_thinking":false}

Benchmarks

These benchmarks were run on an RTX 5090 (limited to 480 W) using llama-cli (b9775) with the CUDA driver and a prompt to generate an Ansible playbook.

Serving with llama.cpp

It has a max context size of 262,114. This can be served using:

llama-server \
-hf CodeFault/Nvidia-Qwen3.6-27B-NVFP4-GGUF \
-hff Nvidia-Qwen3.6-27B-NVFP4-A.gguf \
--temp 0.6 \
--top-p 0.95 \
--top-k 20 \
--repeat-penalty 1.1 \
--spec-type draft-mtp \
--spec-draft-n-max 3