Qwen3.8-Flash-Next-DGX-Spark-Optimized-Recipe

Qwen3.8-Flash-Next on a single DGX Spark — optimized deployment recipe (v5)

A personal, measured-on-one-machine deployment recipe for serving Qwen/Qwen3.8-Flash-Next on a single NVIDIA DGX Spark (GB10) with SGLang. It is not a benchmark leaderboard claim and not "the fastest possible"; it is what is verifiably running on one GB10, with every patch, script, and check needed to reproduce it from upstream weights.

Verified: 2026-09-13 · SGLang 00143e9c23aee2dead5e6fe217bda4fa8739cb92 (nightly-dev-cu13-20260911) · runtime image qwen38-flashnext-sm121-hybrid-sharp:00143e9c-hc2, built by bounded-ple/nightly/01c-build-sglang-nightly.sh. The v4 lineage (d91c3682-hc1) is still in the repo as the stable fallback.

中文摘要：单台 DGX Spark（GB10）上实测可复现的 Qwen3.8-Flash-Next 部署方案。 v5 相对 v4：基线迁到 sglang nightly 00143e9c（模型支持已并入上游，radixark fork overlay 不再重放）、draft KV 转 FP8、KV 池 1.5M、mamba 槽位/检查点参数 定型、bs=4 decode-graph 损坏的规避、KDA SM121 QSA 打包变长 kernel、 ple-graph-fix7 与 chat-error-log overlay。第一次上手请先读 docs/quickstart-zh.md（小白指引）。

What you get

Architecture

graph TD
  A[nvidia/Qwen3.8-Flash-Next-NVFP4<br/>official quantized tree] -->|FP4 experts + FP8 mtp + FP8 PLE| C[HybridSharp-nv target tree<br/>08-merge, hardlinks only]
  B[RadixArk Hybrid-Sharp delta<br/>FP8 blockwise dense bypass] -->|300 FP8 weight pairs| C
  A -->|mtp.* FP8 tensors| D[qwen38-draft-mtp-fp8 draft tree<br/>07-prepare]
  C --> E[SGLang server @ 00143e9c<br/>--quantization modelopt_fp4<br/>qc resolves to modelopt_mixed]
  D --> E

Three quantization domains, one process:

Key runtime facts (all asserted at deploy time by the 05 deploy script):

• quantization_config.quant_method=modelopt_mixed in the target tree makes SGLang resolve ModelOptMixedPrecisionConfig while the CLI stays --quantization modelopt_fp4 (legacy-compatible entry).
• The draft is loaded from its own tree with --speculative-draft-model-path /model-draft --speculative-draft-model-quantization fp8. This is required: _mtp_quant_config drops quantization for drafts that inherit a modelopt_mixed target, so an inherited draft silently becomes BF16.
• Fp8MoEMethod has no cutlass runner upstream (TODO(cwan)); without --speculative-moe-runner-backend triton the draft crashes during CUDA-graph capture with 'Fp8MoEMethod' object has no attribute 'runner'. The bundled fp8.py overlay adds the same fallback as a second line of defense.
• YaRN is injected via --json-model-override-args into text_config (factor = ctx/native, mrope_section [11,11,10], partial 0.25, no original_max_position_embeddings key on purpose).
• The nightly base already carries native qwen4_exp / MTP / PLE / GDN model support (upstream #37500 and the radixark merge PRs), so the v4-era qwen4_exp.py fork overlay is not replayed on this lineage.

Image lineage (nightly, v5)

lmsysorg/sglang:nightly-dev-cu13-20260911-00143e9c
 └─ nightly/01c step 1 → qwen38-flashnext-sm121-community:00143e9c
      · SM121 QSA varlen path + NVFP4 KV method (community layer)
    └─ step 2 → qwen38-flashnext-sm121-bounded-ple:00143e9c
      · rust io_uring O_DIRECT storage extension (rust/ ships the source)
      · build-time O_DIRECT smoke test must pass
    └─ step 3 → qwen38-flashnext-sm121-hybrid-sharp:00143e9c-hc2
      · one moe_hook exact-anchor gate (NVFP4 x flashinfer_cutlass)
      · labels prove all three layers; anchor drift fails the build

nightly-audit.sh re-checks every patch anchor against the running image; exit 1 = an anchor drifted and a layer needs re-anchoring. The runtime overlays are regenerated from the image by nightly/regenerate-overlays.sh into build-nightly/; build-nightly2/ is the versioned, live overlay tree that 05-deploy-hybrid-sharp-nightly.sh mounts.

Upstream sources (download these; this repo ships no weights)

Attribution: this recipe derives from the upstreams above plus SGLang (Apache-2.0). The FP8 dense bypass in the merged tree comes from the Hybrid-Sharp delta; all FP4/FP8 expert bytes come from the NVIDIA tree. The build*/kda_overlay QSA kernel derives from the KDA-1.5 winning submission (see kda_kernels/README.md for the byte-exact provenance hash). Nothing here re-quantizes weights.

Quickstart (one-key)

export MODEL_DIR=$HOME/ai-models REPO_DIR=$HOME/qwen38-flash-next-one-dgx-spark
# 0. prerequisites: DGX Spark (GB10, sm121), docker with nvidia runtime,
#    huggingface_hub CLI logged in, ~250 GB free.
hf download nvidia/Qwen3.8-Flash-Next-NVFP4 --repo-type model --local-dir $MODEL_DIR/Qwen3.8-Flash-Next-NVFP4-hf
#    (RadixArk delta: obtain from its upstream repo per its access terms)
# 1. build the two model trees (hardlinks; needs both downloads present)
python3 build/modelopt_mixed/08-merge-hybridsharp-nv.py
python3 build/modelopt_mixed/07-prepare-draft-fp8.py \
  $MODEL_DIR/Qwen3.8-Flash-Next-NVFP4-hf $MODEL_DIR/qwen38-draft-mtp-fp8
# 2a. nightly lineage (v5, what is running):
bash bounded-ple/nightly/01c-build-sglang-nightly.sh && bash bounded-ple/nightly/regenerate-overlays.sh
# 2b. or stable lineage (v4): bash bounded-ple/01-build-bounded-ple.sh && bash bounded-ple/01b-build-hybrid-runtime.sh
# 3. deploy (idempotent; fingerprinted; auto-rollback on failed health)
bash bounded-ple/hybrid-start-nv-nightly.sh    # nightly entry
# bash bounded-ple/hybrid-start-nv.sh          # stable entry

Manual flow and what each script does

Server flags that matter (nightly v5 defaults)

--context-length 524288 --max-total-tokens 1500000     # effective 1,499,968
--kv-cache-dtype fp8_e4m3 --speculative-draft-kv-cache-dtype fp8_e4m3
--page-size 64 --chunked-prefill-size 1024 --max-prefill-tokens 2048
--prefill-max-requests 1 --max-running-requests 8 --mem-fraction-static 0.95
--mamba-ssm-dtype bfloat16 --mamba-radix-cache-strategy extra_buffer
--mamba-track-interval 64   MAX_MAMBA=40   MAMBA_MAX_STATES=4
--cuda-graph-bs-decode 1 2 3 4 5 6 7 8 --disable-prefill-cuda-graph
--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1
--speculative-num-draft-tokens 4 --speculative-attention-mode decode
--attention-backend flashinfer --decode-attention-backend trtllm_mha
--moe-runner-backend flashinfer_cutlass --quantization modelopt_fp4
--disable-shared-experts-fusion

Two gotchas that cost real debugging time:

• The fingerprint does not cover max_total_tokens or mem_fraction - changing those needs FORCE_REDEPLOY=1, otherwise the deploy silently no-ops.
• --max-total-tokens above the memory ceiling is silently clamped, no error. The only truth is the startup log: KV Cache is allocated ... #tokens: (printed twice: target, then draft).

Verification checklist (grep the server log)

Details per optimization: docs/verification.md. Memory math: docs/memory-budget.md. What broke and how it was fixed: docs/failure-history.md. First time on this machine? docs/quickstart-zh.md (中文小白指引).

What changed since v4

1. Baseline moved to sglang nightly 00143e9c (2026-09-11). Native qwen4_exp/MTP/PLE/GDN support landed upstream (#37500 + radixark merge PRs), so the whole fork overlay is gone; only the SM121 community layer, the bounded-PLE rust ext, and one moe_hook gate remain. Every patch layer is exact-anchor re-anchored; drift fails the build and nightly-audit.sh reports it.
2. Draft KV is FP8 (--speculative-draft-kv-cache-dtype fp8_e4m3). An 8/29 attempt "failed" - that was a build bug (draft runner read the process-global dtype bag), not physics; fixed upstream since. accept_len 2.98, zero harm, and it buys the 1.5M pool (13.35 -> 12.4 KB/token).
3. KV pool 1.11M -> 1.5M. The 9/10 cold-session incident was KV-pool LRU thrash between concurrent long sessions (461K evicted tokens, 225 eviction passes), not mamba tombstones. Pool >= sum of live sessions is a hard constraint; 1.5M is the floor for ~1.1M resident, not headroom.
4. Mamba sizing: MAX_MAMBA=40 slots (peak use 4-8; constraint slots//4 >= maxrun), MAMBA_MAX_STATES=4 per path (idle sessions were going cold as interior checkpoints crowded out session tails), --mamba-track-interval 64 - raising it coarsens the runtime checkpoint grid (lcm) and produced stable semantic garbage at 2-way long-context concurrency.
5. bs=4 decode-graph corruption worked around (upstream #37111, open): the largest captured decode shape replays with whole-buffer row slices and silently degenerates to one repeated token. Fix: capture every shape up to the real ceiling (max_mamba // (1+draft_tokens) = 8) so the victim shape is essentially never replayed. Dropping a shape only moves the victim.
6. KDA SM121 QSA packed-varlen decode kernel in build*/kda_overlay (2.07x geomean vs Triton fallback; TP1 BK32 threshold 12 after the bs=4 token-0 corruption was traced to graph coverage, not dispatch).
7. ple-graph-fix7 overlays (qwen4_ple_nvme.py + full_cuda_graph_backend.py) make PLE gather correct under full CUDA graphs; markers are preflighted and fingerprinted.
8. serving_chat.py chat-error-log overlay: chat errors are logged with the request instead of vanishing into a bare 500.
9. rust/ source is now shipped - v4's Dockerfiles referenced it but the repo omitted it; the build is finally self-contained.

Known limitations

• Single-machine personal recipe; numbers are from one GB10, one session mix.
• The bs=4-class graph defect is upstream-open (#37111); the mitigation is shape-list sizing, and a future fix may want MAX_MAMBA >= 45 for a true sentinel shape.
• The triton draft-MoE autotune workspace costs a few percent of the KV pool; FP8 draft KV more than pays it back.
• FP8 KV runs with scale 1.0 (no calibration scales shipped); the warning is expected.
• SM121-specific: the cutlass pin and KDA/QSA kernels are for GB10; other Blackwell parts need re-derivation of --moe-runner-backend.
• The RadixArk delta is required by the merge; if you cannot obtain it, run the pure NVIDIA tree with --quantization modelopt_fp4 and drop the FP8 dense bypass (acceptance will differ).
• Upstream sglang PR #36567 (stream PLE from NVMe) is still open; until it merges, the bounded-PLE rust extension stays load-bearing.

Repo layout

bounded-ple/   deploy scripts, Dockerfiles, PLE+chat overlays, rust/ source,
               nightly/ build chain, upgrade/ merge tools, checks
build/         d91c3682-lineage runtime overlays + tree builders 06/07/08
build-nightly2/  live (00143e9c-lineage) runtime overlay tree
docs/          failure-history.md, verification.md, memory-budget.md,
               quickstart-zh.md (中文小白指引)
nightly-audit.sh  anchor-drift audit for the running container

License & disclaimer

Code in this repo: MIT (see LICENSE). Model weights are not included and remain under the licenses of Qwen/Qwen3.8-Flash-Next, nvidia/Qwen3.8-Flash-Next-NVFP4, and the RadixArk delta. This is an independent personal optimization effort, not affiliated with NVIDIA, RadixArk, or the Qwen team. Verify on your own hardware before trusting any number here.