Step 3.7 Flash GGUF
My custom IQ4_XS GGUF quantization for stepfun-ai/Step-3.7-Flash
I've also modified the chat template also adds a preserve_thinking option,
which preserves thinking across user turns and can improve the experience when
prompt processing speed is a bottleneck.
Quant Recipes
| Recipe | Quant Size | Default type | Tensor-specific overrides |
|---|---|---|---|
IQ4_XS | 101784.88 MiB (4.34 BPW) | Q6_K | ffn_down_exps=iq4_xs, ffn_gate_exps=iq4_xs, ffn_up_exps=iq4_xs |
Related Files
| File | Description |
|---|---|
Step-3.7-Flash-MTP-Q8_0.gguf | Q8_0 MTP weights |
Step-3.7-Flash-mmproj-BF16.gguf | BF16 multimodal projector |
Step-3.7-Flash-mmproj-F16.gguf | F16 multimodal projector |
Step-3.7-Flash-mmproj-Q8_0.gguf | Q8_0 multimodal projector |
Usage
Here's an example script:
#!/bin/sh -e
model="./IQ4_XS/Step-3.7-Flash-IQ4_XS-00001-of-00004.gguf"
mmproj="./Step-3.7-Flash-mmproj-Q8_0.gguf"
mtp=./Step-3.7-Flash-MTP-Q8_0.gguf
ctx=262144
parallel=1
ctx_size=$((ctx * parallel))
reasoning_budget_message="...
Actually, I will stop now.
Let me provide the user with a comprehensive answer."
llama-server --no-mmap --no-warmup --model $model --mmproj $mmproj \
--ctx-size $ctx_size -np $parallel --temp 1.0 --top-p 0.95 \
--repeat-penalty 1.0 --presence-penalty 0.0 \
--reasoning-budget-message "$reasoning_budget_message" \
--reasoning-preserve \
--spec-type draft-mtp -md $mtp --spec-draft-n-max 3 --spec-draft-p-min 0.65 \
-ctxcp 8 --checkpoint-min-step 512 \
--cache-ram 4096