Qwen3.5-9bxOpus-4.7-sonic-GGUF : GGUF
A distilled 9B code/reasoning model in GGUF format, optimized for local inference via llama.cpp.
Quick Start
llama-cli -hf nphearum/Qwen3.5-9bxOpus-4.7-sonic-GGUF --jinja
Server Launch
llama-server \
-m Qwen3.5-9bxOpus-4.7-sonic-GGUF.Q4_K_M.gguf \
--port 8001 --alias qwen3.5-9b-opus \
-c 65536 -n 8192 \
--temp 0.6 --top-p 0.95 --top-k 40 --repeat-penalty 1.05 \
--flash-attn on --ctk q8_0 --ctv q8_0 \
--jinja --chat-template-kwargs '{"enable_thinking": true}' \
-ngl -1
Recomment configure
Using mmproj is more like to Multi-Token Prediction(MTP) -- reason for support text, image, video, audio and pdf, is an advanced training and inference technique for memory and cache.
I recomment to use --ctk q8_0 --ctv q8_0 for better cache and load.
llama-server \
-m Qwen3.5-9bxOpus-4.7-sonic-GGUF.Q4_K_M.gguf \
--port 53281 \
-c 131072 \
--parallel 1 \
--flash-attn on \
--no-context-shift \
-ngl -1 \
--jinja \
--chat-template-kwargs "{\"enable_thinking\": true}" \
--mmproj Qwen3.5-9bxOpus-4.7-sonic-GGUF.BF16-mmproj.gguf
Preset Configs
| Mode | Flags |
|---|---|
| Coding | --temp 0.4 --top-p 0.9 --top-k 50 --repeat-penalty 1.1 |
| Reasoning | --temp 0.6 --top-p 0.95 --top-k 40 |
| Low VRAM | -c 32768 -n 4096 --flash-attn off -ngl 20 |
Key Specs
- Base: Qwen3.5 9B · Format: Q4_K_M GGUF
- Context: 32K–64K practical · Output: up to 8K tokens
- KV cache: 8-bit (q8_0) · GPU: full offload (
-ngl -1) - Thinking mode: optional — improves reasoning, adds latency
Caveats
- Quality drops beyond 64K context
- 9B class — sensitive to sampling parameters
- Thinking mode can produce unstable output if misaligned with training