Qwen3.8-Flash-Next-WinVulkan-GGUF

Qwen3.8-Flash-Next (Qwen 3.8 Next Flash) — Windows Vulkan 全适配包 / Windows Vulkan Full Deployment Kit

EN | Complete local deployment kit for Qwen3.8-Flash-Next — a ~180 B model: 125 B transformer (6 B activated per token) + a 51 B n-gram embedding table + a 4 B MTP draft (ROCmFP4-FAST-v2-ple16 fork quantization) — running on Windows + Vulkan — AMD Ryzen AI MAX+ 395 (ROG Flow Z13, 128GB LPDDR5X, 96GB VRAM carve-out). Includes MTP speculative decoding, PLE n-gram table disk-offload, custom llama.cpp fork binaries, startup scripts and a custom WebUI. This build is original adaptation work — no off-the-shelf guide exists.

中文 | Qwen3.8-Flash-Next（总参约 180B = 主干 125B / 每 token 激活 6B + n-gram 表 51B + MTP 草稿 4B；ROCmFP4-FAST-v2-ple16 fork 专用量化）在 Windows + Vulkan 上的完整本地部署方案 — AMD Ryzen AI MAX+ 395（ROG Flow Z13 / 128GB LPDDR5X / 96GB 显存 carve）。含 MTP 投机解码、PLE 表磁盘化、定制 llama.cpp fork 二进制、启动脚本与定制 WebUI。本包为原创适配，网上无现成方案。

📐 Model facts / 模型参数 — total ~180 B parameters: 125 B transformer (6 B activated per token, 48 layers, hidden 2560, 512 MoE experts with 10 routed + 1 shared, hybrid SSM/full-attention every 4th layer, 262144-token context) + 51 B n-gram embedding table (per_layer_token_embd.weight, [160 × 320001536]) + 4 B MTP draft. Measured from the GGUF headers: main file 176.94 B tensor parameters (125.74 B backbone + 51.20 B n-gram table), MTP draft 3.88 B, mmproj 0.45 B — matching upstream's "125B with 6B activated, plus 51B n-gram embedding and 4B MTP". ⚠️ 87.06 GiB / 93.48 GB is the size of the main quant file at 4.23 bpw — it is not a parameter count. Earlier revisions of this card said "87B MoE", which mixed up file size and parameter count. 中文：总参约 180B = 主干 125B（每 token 激活 6B）+ n-gram 表 51B + MTP 草稿 4B；GGUF 头实测主模型 176.94B 张量参数、草稿 3.88B、mmproj 0.45B，与上游口径一致。87.06 GiB / 93.48 GB 是 4.23 bpw 量化文件的体积，不是参数量（旧版卡片写 "87B MoE" 属单位混用，已更正）。

🔧 Fork binaries, source patches, scripts & tools → TonyMFJ/Qwen3.8-Flash-Next-WinVulkan 🔧 Fork 二进制、源码补丁、脚本与工具 → TonyMFJ/Qwen3.8-Flash-Next-WinVulkan

---

📦 Files / 文件清单

📏 Repository total: 119.57 GB (111.35 GiB) of downloads. A single run needs ≈ 97.17 GB (90.5 GiB) of weights — main file 93.48 + mmproj 0.90 + draft 2.79; the sidecar is the same n-gram table in a second layout, counted once. Sizes are decimal GB as shown in the HF file browser; GiB in brackets for du-style comparisons. 📏 仓库合计 119.57 GB (111.35 GiB)（下载总量）；单次运行实际占用约 97.17 GB (90.5 GiB)（主模型 + mmproj + 草稿；侧车是同一张 n-gram 表的另一种布局，只算一次）。括号内为 GiB（与本地 du/资源管理器口径一致）。

💡 Equivalent alternative to the sidecar: convert the main GGUF to joined layout locally with tools/gguf_split_ple_heads.py (join) + --ngram-on-disk. 💡 侧车的等价方案：用 tools/gguf_split_ple_heads.py（join 模式）把主模型本地转成 joined 布局，再配合 --ngram-on-disk。

⚡ Measured performance / 实测性能（reference machine）

⚠️ All figures below were measured with the APU package power limited to 80 W (AMD Ryzen AI MAX+ 395, 128 GB LPDDR5X, 96 GB VRAM carve-out, Windows 11, Vulkan backend). A different power profile gives different numbers — check the wattage before comparing.

Rows come from different sessions/flag sets — treat them as order-of-magnitude references. Pin the power profile, context length, KV placement and MTP state before comparing.

Full measured curve — huihui-abliterated UD-Q4_K_XL build (111.3 GB / 103.7 GiB, 4 shards)

Same machine and power limit, config -c 102400 -b 2048 -ub 2048 -ctk q8_0 -ctv q8_0 (KV cache in VRAM) + MTP draft (adaptive n-min 2 / n-max 4) + PLE n-gram table on disk:

• PLE n-gram table (28.8 GB) stays on disk → peak VRAM 90.4 / 96 GB
• MTP is the decode lever: without the draft model decode falls to ≈ 6–8 t/s
• End-to-end, 90K prompt + 500 output tokens: 429 s with MTP (410 prefill + 19 decode) vs 453 s without (368 + 85)
• Acceptance above is inflated by highly repetitive test text; on natural text it measures 59–83 % (decode ≈ 21–35 t/s)
• MTP costs ≈ 10 % prefill: the draft context forces ubatch 4096 → 2048 (3072/4096 both fail to allocate)

同机同功耗下的完整曲线：prefill 402.8 → 220.6 t/s（4.5K → 90.4K），decode 22–35 t/s（含 MTP）； 模型本体 huihui UD-Q4_K_XL 4 段共 111.3 GB，PLE 表 28.8 GB 走磁盘，显存峰值 90.4/96 GB。

Environment & reproduction: driver 32.0.31032.1003, w64devkit 2.9.1 / GCC 16.2.0, CMake 4.4.0, Vulkan SDK headers 1.4.309 (MinGW Makefiles). Build commands, model layout, verification steps and the pitfall list live in ENVIRONMENT.md.

Tuning levers & pitfalls: MTP acceptance, KV placement (host vs VRAM, ~2x long-context decode), the ubatch ceiling and the misleading “unused tensor” PLE warning → TUNING_NOTES.md.

HIP / ROCm path — negative result: we also brought this model up on ROCm 7.2 + HIP. The PLE disk-offload port works (418.7 t/s prefill at 2.6K), but with head_dim = 256 FlashAttention falls back to the slow tile kernel and upstream's MMA kernel for head_dim > 128 is an empty stub — HIP never beats Vulkan. Full write-up, root cause and artifacts (patched backend, port patch, official DLL mirror) → HIP_NOTES.md · release assets

中文：HIP / ROCm 路径也跑通过（PLE 磁盘化移植成功），但 head_dim=256 让 FlashAttention 掉进退化路径、 上游 MMA 对 head_dim>128 是空壳 kernel，快不过 Vulkan，详见 HIP_NOTES.md 与 Release 附件。

🚀 Key launch flags / 关键启动参数

llama-server -m <ple16.gguf> --model-ple <ple_sidecar_joined.gguf> \
  --mmproj <mmproj-f16.gguf> -md <mtp-Q4_K_M.gguf> \
  -c 102400 --ubatch-size 2048 -ngl 99 --load-mode auto ...

• EN | --load-mode auto is REQUIRED on this fork/Windows (never none). PLE table on disk ≈ −37% speed but frees ~22 GB VRAM for 100K+ context. Full-VRAM mode reaches ~28 t/s (TG) with MTP acceptance ≈ 67%.
• 中文 | 本 fork 在 Windows 上必须用 --load-mode auto（none 会崩）。PLE 表磁盘化约损失 37% 速度，换来约 22GB 显存用于 100K+ 上下文。全显存模式 TG 约 28 t/s，MTP 接受率约 67%。

📚 Sources / 来源

• Base quant / 基础量化: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF (ple16)
• MTP draft / 草稿模型: quimmedes/Qwen3.8-Flash-Next-MTP-GGUF
• Sidecar / 侧车: generated locally with this fork's tools / 用本 fork 工具本地生成

📜 Restore walkthrough / 换机恢复指南

See GitHub repo README_恢复指南.md and MODELS.md — unzip, drop models, run one script. 见 GitHub 仓库的 README_恢复指南.md 与 MODELS.md —— 解压、放模型、跑一个脚本即可。