Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill

Lynn Agent edge runtime

This 35B-A3B distillation is the high-end local orchestrator for Lynn Agent — the sparse (3B-active) sibling of the 27B Dense distill. On 32 GB+ VRAM / unified-memory machines Lynn can run it as the task orchestrator (decompose → delegate → verify); 24 GB machines use the 27B path instead.

• Download Lynn Agent: GitHub Releases v0.85.6
• GGUF (edge): Hugging Face 35B GGUF / ModelScope 35B GGUF
• FP8 (concurrent serving): Hugging Face 35B FP8
• Recommended quant: Q5_K_M imatrix + native MTP (single-stream), or FP8 for concurrent serving.

🇬🇧 English · 🇨🇳 中文 ⬇️

🇬🇧 English

On Qwen3.6-35B-A3B (MoE, 3B active), we use LoRA to distill the way DeepSeek-V4-Pro reasons (with thinking-on) plus its agentic behavior — purpose-built as a fast task orchestrator (decompose → delegate → verify) for Lynn Agent.

This is the MoE counterpart of the 27B Dense sister model: same R6000 GPU, same teacher, same recipe, on a sparse architecture. A native MTP (nextn) head is welded on for single-stream acceleration.

⚠️ Distilling a thinking style ≠ distilling knowledge/capability: the goal is "learn how to reason and how to converge", not to inject knowledge or raise the capability ceiling.

Training details

• Base: Qwen3.6-35B-A3B (MoE, 3B active, BF16 base)
• Method: LoRA, r = 64, α = 128, dropout = 0.05, targets = all attention + MLP projections
• Optim: paged_adamw_8bit, cosine LR, warmup 0.03, ~1 epoch
• Teacher: DeepSeek-V4-Pro (thinking-on + agentic)
• Data: ~1,842 distillation samples (lynn_prod spec). Trajectories = DS-V4-Pro multi-step reasoning under thinking-on (<think>) + ReAct-style tool calls (think one step → call one tool → observe → loop).
  • The tool "execution results" are SIMULATED, not actually run — each result line is improvised by a small fast model (DeepSeek-V4-Flash) role-playing the runtime, not real sandbox execution.
  • Training masks those fabricated results — the model learns only "how to think / how to call tools", never the made-up outputs (so it doesn't learn to fabricate tool returns).

Attribution (the method is not original — it is a combination of published techniques)

• ReAct (interleaved reasoning + acting): Yao et al., 2022, arXiv:2210.03629 (ICLR 2023)
• STaR (bootstrapping reasoning traces): Zelikman et al., 2022, arXiv:2203.14465
• Self-Instruct / Baize self-chat: Wang et al., 2022; Xu et al., 2023, arXiv:2304.01196
• AgentTuning: Zeng et al., 2023, arXiv:2310.12823
• ToolBench / ToolLLM: Qin et al., 2023, arXiv:2307.16789
• DeepSeek-R1 reasoning distillation: DeepSeek-AI, 2025, arXiv:2501.12948

Evaluation — distill vs base, same-spec (Q4_K_M imatrix GGUF + MTP), thinking-on

Quantization parity: this model and the base are both Q4_K_M (imatrix) GGUF, same-spec — the only variable is "distilled or not", so the Δ is cleanly attributable to distillation.

Reading: hard reasoning improves markedly (GPQA +7.6pp) and the model becomes decisive — the end-to-end orchestration loop is 2.3× faster purely from fewer tokens to a decision (single-stream TPS is identical to base, ~224 t/s on R6000 / llama.cpp), and GPQA non-terminating "empty" answers collapse from 12 → 1 (the convergence / 收口 effect). Honest caveat: unlike the 27B Dense (which nudged MMLU up), this MoE dips MMLU ~1.2pp — a sparse model has less spare capacity, so the distillation costs a sliver of knowledge breadth. The trade is GPQA + decisiveness for a small MMLU dip.

MTP — native nextn speculative decoding

Two MTP paths: GGUF via llama.cpp (--spec-type draft-mtp, below); BF16 / FP8 safetensors via vLLM / SGLang (--speculative-config '{"method":"mtp","num_speculative_tokens":3}') — all of BF16, FP8 and the four GGUF quants bundle the native nextn head.

Per-quant GGUF MTP speed (mainline llama.cpp, --spec-type draft-mtp --spec-draft-n-max 3, single-stream on DGX Spark GB10, server multi-request avg):

• safetensors (vLLM / SGLang): SGLang-measured accept 0.71–0.87; FP8 single-stream 51 → 65 t/s (~1.25×).
• ⚠️ A3B is a sparse (MoE) model — single-stream MTP gain is bounded by the density law and is far below the dense 27B's ~2×. The slower the base quant, the larger the relative gain (Q8_0 +63%).
• ⚠️ LM Studio / Ollama / Jan and most GUIs do not support the embedded nextn MTP (their "speculative decoding" attaches a separate draft model) — the GGUF loads and runs in base mode (works, no MTP speedup). MTP only applies with mainline llama.cpp CLI --spec-type draft-mtp.

📏 GB vs GiB. File sizes below are decimal GB (10⁹ bytes), but GPU VRAM is built in GiB (2³⁰) and merely labeled "GB" — so a "32 GB" card is really 32 GiB ≈ 34.4 GB. Rule of thumb: a file runs with no CPU offload when its GiB size < the card's nominal "GB" number.

Eval protocol

thinking-on; temperature 0.6 / top_p 0.95 (required for thinking models — greedy loops to death); max_tokens 32768; read-timeout ≥ 2400s. The same spec is applied to every compared model.

Limitations

• Distills thinking style, not capability: black-box SFT cannot raise the knowledge ceiling; MMLU dips slightly (−1.2pp) on this MoE.
• Built as an orchestrator, not a broad knowledge model — its strength is decompose / delegate / verify + convergence, not raw breadth.
• Orchestration-success / false-verify are judge-subjective trend metrics; the hard numbers are GPQA + the convergence (empty 12→1) data.

Files

• *.safetensors — BF16 (SGLang / vLLM / transformers), ~69 GB
• gguf/*-imatrix.gguf — Q4_K_M (~21 GB) / Q5_K_M (~25 GB) / Q6_K (~29 GB) / Q8_0 (~37 GB) / F16 (~69 GB), all with native MTP
• gguf/*-mmproj-F16.gguf — vision projector (~1 GB; image input)
• FP8 build: see the FP8 repo (concurrent serving)

Inference

• thinking-on, temperature=0.6, top_p=0.95 (never greedy).
• llama.cpp: llama-server -m *-Q5_K_M-imatrix.gguf --spec-type draft-mtp --spec-draft-n-max 3 --jinja
• vLLM / SGLang: load BF16 or FP8 + --speculative-config '{"method":"mtp","num_speculative_tokens":3}'.

Claude Code (experimental)

Claude Code is not a local GGUF/HF loader — it talks to an Anthropic-compatible /v1/messages backend and needs reliable tool calling, so you cannot point it at a repo id directly. vLLM path (BF16 / FP8 safetensors):

vllm serve /path/to/model \
  --served-model-name qwen36-35b \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml

Set Claude Code's model name to the --served-model-name (no slashes) and ANTHROPIC_BASE_URL to your vLLM endpoint. GGUF path: LM Studio 0.4.1+ loads the GGUF and exposes a built-in Claude Code /v1/messages endpoint. Refs: vLLM Claude Code · LM Studio.

🇨🇳 中文版

在 Qwen3.6-35B-A3B（MoE，3B 激活）上，用 LoRA 蒸馏 DeepSeek-V4-Pro 在「思考开启（thinking-on）」时的思维方式 + agentic 行为 —— 专门做 Lynn Agent 的快速任务编排器（拆分 → 分派 → 验收）。

这是 27B Dense 姊妹版的 MoE 复现：同一台 R6000、同一 teacher、同一套配方，换到稀疏架构。并焊了**原生 MTP（nextn）**头做单流加速。

⚠️ 蒸思维方式 ≠ 蒸知识/能力：目标是「学会怎么想、怎么收口」，不是蒸知识或扩能力上限。

训练配置（如实披露）

• 基座：Qwen3.6-35B-A3B（MoE，3B 激活，BF16 基座）
• 方法：LoRA，r=64 / α=128 / dropout 0.05，targets = 全部 attention + MLP 投影
• 优化：paged_adamw_8bit，cosine LR，warmup 0.03，~1 epoch
• Teacher：DeepSeek-V4-Pro（thinking-on + agentic）
• 数据：~1842 条蒸馏样本（lynn_prod 口径）。轨迹 = DS-V4-Pro thinking-on（<think>）多步推理 + ReAct 式工具调用（想一步 → 调一个工具 → 观察 → 循环）。
  • 工具「执行结果」是模拟的、不是真跑 —— 每条结果由小快模型（DeepSeek-V4-Flash）扮演 runtime 即兴生成，非真实沙箱执行。
  • 训练 mask 掉这些伪造结果 —— 模型只学「怎么想 / 怎么调工具」，不学编造的输出（避免学会瞎编工具返回值）。

方法非自创，是公开技术的组合（如实归因）

• ReAct（推理+行动交织）：Yao 等 2022，arXiv:2210.03629（ICLR 2023）
• STaR：Zelikman 等 2022，arXiv:2203.14465
• Self-Instruct / Baize self-chat：Wang 等 2022；Xu 等 2023，arXiv:2304.01196
• AgentTuning：Zeng 等 2023，arXiv:2310.12823
• ToolBench / ToolLLM：Qin 等 2023，arXiv:2307.16789
• DeepSeek-R1 推理蒸馏：DeepSeek-AI 2025，arXiv:2501.12948

评测 —— 蒸馏 vs 原版，同规格（Q4_K_M imatrix GGUF + MTP），thinking-on

量化对齐：本模型和原版都是 Q4_K_M（imatrix）GGUF、同规格，唯一变量是「是否蒸馏」，Δ 干净归因于蒸馏本身。

解读：硬推理显著提升（GPQA +7.6pp），且模型变果断 —— 端到端编排循环**快 2.3×**纯粹来自更少 token 就给出决策（单流 TPS 与原版相同，R6000 / llama.cpp ~224 t/s），GPQA 不收口的「空答」从 12 → 1（收口效果）。如实说明:与 27B Dense（MMLU 还微升）不同,这个 MoE 的 MMLU 微降 ~1.2pp —— 稀疏模型富余容量更少,蒸馏会挤掉一点知识广度。换来的是 GPQA + 果断收口,代价是 MMLU 略降。

MTP —— 原生 nextn 投机解码

两条 MTP 通路：GGUF 走 llama.cpp（--spec-type draft-mtp，见下）；BF16 / FP8 safetensors 走 vLLM / SGLang（--speculative-config '{"method":"mtp","num_speculative_tokens":3}'）—— BF16、FP8 与四档 GGUF 均已焊原生 nextn 头。

各量化档 GGUF MTP 速度（mainline llama.cpp，--spec-type draft-mtp --spec-draft-n-max 3，DGX Spark GB10 单流，server 多请求均值）：

• safetensors（vLLM / SGLang）：SGLang 实测 accept 0.71–0.87；FP8 单流 51 → 65 t/s（~1.25×）。
• ⚠️ A3B 是稀疏（MoE）模型 —— 单流 MTP 增益受密度定律限制，远低于 dense 27B 的 ~2×。base 越慢（档位越大）相对增益越大（Q8_0 +63%）。
• ⚠️ LM Studio / Ollama / Jan 等 GUI 暂不支持嵌入式 nextn MTP（它们的"投机解码"挂的是独立 draft 模型）—— GGUF 加载后以 base 模式运行（可正常使用，无 MTP 加速）；MTP 加速仅在 mainline llama.cpp CLI 启用 --spec-type draft-mtp 时生效。

📏 GB vs GiB：下方体积为十进制 GB（10⁹ 字节）；显卡显存按 GiB（2³⁰） 造却标 "GB"，故 "32 GB" 卡实为 32 GiB ≈ 34.4 GB。规则：文件的 GiB 数 < 卡标称 "GB" 数即可不 offload。

评测口径 / Eval protocol

thinking-on；temp 0.6 / top_p 0.95（thinking 模型必需，greedy 会重复死循环）；max_tokens 32768；read-timeout ≥ 2400s。同口径作用于所有对比模型。

局限 / Limitations

• 蒸思维方式、不蒸能力：黑盒 SFT 抬不高知识上限；本 MoE 的 MMLU 微降（−1.2pp）。
• 定位是编排器，非博学模型 —— 强在拆分/分派/验收 + 收口，不是知识广度。
• 编排成功/假验证是判官主观的趋势指标；硬数据是 GPQA + 收口（空答 12→1）。

文件 / Files

• *.safetensors —— BF16（SGLang / vLLM / transformers），~69 GB
• gguf/*-imatrix.gguf —— Q4_K_M（~21 GB）/ Q5_K_M（~25 GB）/ Q6_K（~29 GB）/ Q8_0（~37 GB）/ F16（~69 GB），均含原生 MTP
• gguf/*-mmproj-F16.gguf —— 视觉投影（~1 GB；图像输入）
• FP8 版见 FP8 仓（并发 serving）

推理 / Inference

• thinking-on，temperature=0.6, top_p=0.95（切勿 greedy）。
• llama.cpp：llama-server -m *-Q5_K_M-imatrix.gguf --spec-type draft-mtp --spec-draft-n-max 3 --jinja
• vLLM / SGLang：加载 BF16 或 FP8 + --speculative-config '{"method":"mtp","num_speculative_tokens":3}'。

Claude Code（实验性 / experimental）

Claude Code 不是本地 GGUF/HF 加载器 —— 它对接兼容 Anthropic /v1/messages 的后端、且要求可靠的工具调用，不能直接喂仓名/路径。vLLM 路线（BF16 / FP8 safetensors）：

vllm serve /path/to/model \
  --served-model-name qwen36-35b \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml

Claude Code 里模型名填 --served-model-name（不带 /），ANTHROPIC_BASE_URL 指向你的 vLLM 端点。GGUF 路线：LM Studio 0.4.1+ 加载 GGUF，自带 Claude Code /v1/messages 端点。参考 vLLM Claude Code · LM Studio。