Qwen3.5 122B A10B — hybrid INT4 + FP8 (DGX Spark) A hybrid quantized checkpoint of Qwen3.5 122B A10B for bandwidth optimal decode on the NVIDIA DGX Spark (GB10 / SM121, 128 GB unified) under vLLM. Routed experts and attention stay INT4; the BF16 shared experts (dense — read on every token) are replaced with calibrated FP8 from the official FP8 release. Composition Routed experts + attention: INT4 (GPTQ / AutoRound) — from Intel/Qwen3.5 122B A10B int4 AutoRound . Shared experts: FP8 E4M3 block 128 — from Qwen/Qwen3.5 122B A10B FP8 . Embeddings / norms / head: unchanged from the INT4 base. 144 shared expert layers convert BF16→FP8 — an always on bandwidth lever worth +28 % at base / no spec decode on GB10 (28.2 → 36.0 tok/s), diminishing under high acceptance speculative decode (see the project's amortization law). Usage Built for and served by qwen3.5 122B A10B on spark ( profile dense ). It requires that project's vLLM 0.23 hybrid FP8 dispatch patch — an INCConfig.maybe update config override that dispatches Fp8LinearMethod for the FP8 shared expert layers. Stock vLLM will not dispatch the mixed INT4/FP8 scheme. Provenance & credit Recipe (hybrid INT4+FP8, INT8 lm head, MTP) and bu…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy