[!IMPORTANT] MTP Layers Grafted on Unsloth UD XL Quantizations The GGUF files in this repo are Unsloth Dynamic 2.0 XL (UD XL) quantizations of Qwen3.6 35B A3B with Multi Token Prediction (MTP) layers grafted on top . Base quantization: Unsloth UD XL — see Unsloth Dynamic 2.0 GGUFs for benchmarks. MTP layers: Taken from am17an/Qwen3.6 35BA3B MTP GGUF and stored in Q8 0 precision, then merged into the UD XL GGUFs. Why Q8 for MTP? The draft heads are small relative to the base model, so Q8 0 keeps them near lossless while avoiding the overhead of full requantizing the entire stack. convert.py : The script used to graft MTP layers into the UD XL GGUFs. Adapted from this gist. 35BA3B MTP.gguf : The raw Q8 0 MTP layer source file (extracted only) used as input to convert.py . To run these files with MTP you need a custom build of llama.cpp that includes the MTP/speculative decoding support from PR 22673. Follow the steps below. 🐳 Or Use Pre built Docker Images (Recommended) Don't want to build from source? Ready to use Docker images with MTP support are available for multiple backends: Image Backend havenoammo/llama:cuda13 server CUDA 13 havenoammo/llama:cuda12 server CUDA 12 havenoammo…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy