Qwen3.8-Flash-Next-GB10-SGLang-Recipe

Qwen3.8-Flash-Next on one DGX Spark (GB10) — reproducible SGLang recipe

Superseded (2026-10-31). On the same GB10, with the same tests and the same 524k context, pendraga/Qwen3.8-Flash-Next-GB10-vLLM-524k (vLLM + b12x, built on diffbot's FP4-Local recipe) decodes +32 % on code and +23 % on thinking (78 vs 59.4 and 65 vs 52.4 tok/s) and passes long-context checks up to 489k tokens. This SGLang recipe still builds and runs as documented below and remains the deterministic option (bit-identical NLL across runs), but it is no longer maintained.

A buildable, validated fork of single-spark-ai/Qwen3.8-Flash-Next-DGX-Spark-Optimized-Recipe (v4, commit 10e8a24, MIT, © 2026 hashd1ve). The upstream recipe describes an excellent design — NVFP4 routed experts + FP8 blockwise dense bypass + FP8-serialized MTP draft + YaRN×2 to 524K + bounded io_uring PLE — but does not build or run from public sources as published. This fork fixes that, fixes a scheduler crash, re-enables vision, and adds an FR-Spec hot-token map for the draft that brings decode from 48.5 to 60 tok/s on code.

Everything below was run end to end on one GB10 on 2026-09-17. No weights in this repo. Not affiliated with the upstream author, NVIDIA, RadixArk, LMSYS/SGLang or Qwen.

Results (single stream, streaming decode excluding TTFT, 512 tokens, temp 0)

Clean-room re-run from a fresh clone (see below): code 59.1 · prose 40.4 · German 44.9 · thinking 51.6 tok/s. GSM8K (200 problems, thinking on): 191/200 = 95.5 %.

Upstream claims 63.1 tok/s on code; see Why not 63 tok/s.

What was broken upstream, and what this fork does

Removed from the upstream tree (pre-v4 or unused, never exercised here): 02/03/04-*, legacy-start/stop.sh, hybrid-sharp-review.md, hybrid_sharp_tree_check.py. They remain available in the upstream repo.

Architecture

graph TD
  A[nvidia/Qwen3.8-Flash-Next-NVFP4<br/>official quantized tree] -->|FP4 experts + FP8 mtp + FP8 PLE| C[HybridSharp-nv target tree<br/>08-merge, hardlinks only]
  B[RadixArk Hybrid-Sharp delta<br/>FP8 blockwise dense bypass] -->|300 FP8 weight pairs| C
  A -->|mtp.* FP8 tensors| D[qwen38-draft-mtp-fp8 draft tree<br/>07-prepare]
  H[tokenmap/hot_token_map_de_code_65536.pt] -->|draft lm_head rows| E
  C --> E[SGLang server<br/>--quantization modelopt_fp4 → modelopt_mixed<br/>NEXTN 3/1/4, vision on]
  D --> E

Key runtime facts from upstream still hold: the draft must load from its own tree (_mtp_quant_config drops quantization for drafts inheriting a modelopt_mixed target); Fp8MoEMethod has no cutlass runner, hence the triton draft MoE; YaRN is injected via --json-model-override-args without original_max_position_embeddings.

Requirements

• NVIDIA DGX Spark / GB10 (SM121), Docker with GPU access via CDI (--device nvidia.com/gpu=all)
• uv/uvx, Hugging Face CLI login (uvx --from huggingface_hub hf auth login)
• ~250 GB free disk during build (~137 GB downloads, hardlinked trees)
• ~20 GB free RAM for the draft-tree step; stop other large UMA consumers first
• On UMA, running out of memory freezes the node. Startup dips to ~5 GiB available.

Build and deploy

git clone <this repo> qwen38-flash-next-gb10-sglang && cd qwen38-flash-next-gb10-sglang
export MODEL_DIR=$HOME/ai-models

bash scripts/00-download.sh                   # ~137 GB, pinned revisions            -> DOWNLOAD-OK
bash scripts/02-build-trees.sh                # target merge + FP8 draft tree         -> HYBRIDSHARP-NV-OK, DRAFT-OK
bash bounded-ple/01-build-bounded-ple.sh      # pulls base by digest, builds io_uring -> "smoke test: PASS"
bash bounded-ple/01b-build-hybrid-runtime.sh  # server-args gate                      -> "gate: PASS"
bash scripts/03-probe-trees.sh                # CPU probes in image                   -> PROBE-MIXED-OK, PROBE-DRAFT-FP8-OK
bash bounded-ple/hybrid-start-nv.sh           # ~10 min load; health + smoke + rollback trap

Deploy knobs (environment, read by bounded-ple/05-deploy-hybrid-sharp.sh):

Port, bind, key, restart policy, token map, spec and vision knobs are not part of the deploy fingerprint: combine changes with FORCE_REDEPLOY=1.

Crash loop note: with unless-stopped, a server that keeps dying is restarted forever (each start loads 76 GB for ~10 min). Use bash bounded-ple/hybrid-stop.sh.

Verify

Every step above was re-run from a fresh clone of this repo on 2026-09-17, including a byte-identical rebuild of both model trees and a numerically identical deployment (teacher-forced NLL equal to 4 decimals). Full log: results/cleanroom-2026-09-17.md.

python3 tests/crashtest.py                    # prefix hit, 9.6k-token prompt, follow-up — all "OK", correct facts
python3 tests/spec_bench.py mylabel 3         # streaming decode for code / prose / German / thinking
python3 tests/quality.py mylabel              # teacher-forced NLL (exactly reproducible) + task checks
uv run --no-project --with huggingface_hub --with pyarrow \
  python3 tests/gsm8k_eval.py mylabel 200 --thinking       # accuracy sanity check
QWEN_BASE_URL=http://host:8000 python3 tests/crashtest.py   # remote server

Server log lines to expect:

Status of the stock upstream path

As of SGLang v0.5.21 every patch in this fork exists upstream, and scripts/10-deploy-upstream.sh deploys the same trees on the stock image with no overlays. It boots, but the merged Hybrid-Sharp tree emits garbage there — isolated to the tree itself, not to speculation, the token map, the PLE table or the quantization flag. The plain NVIDIA NVFP4 checkpoint does work upstream, at 28 tok/s versus 59 tok/s here, because upstream cannot read the FP8 dense bypass that makes this recipe fast. The pinned 4ccff141 + overlays therefore remain the supported path: docs/gb10-findings.md §11.

Client pitfall: truncated tool calls

Long-context fidelity is clean (verbatim lines and retrieval exact at 172k prompt tokens, tests/longctx_fidelity.py), and identifier fidelity is 30/30. But a file-writing agent needs generous max_tokens: a full single-file HTML page is ≈2,300 completion tokens inside the tool-call argument. At max_tokens=2048 the server returns finish_reason=length and the truncated arguments still parse as valid JSON, so a client writes a broken file and the model loops trying to repair it. Set max_tokens ≥ 16k in your client and reject tool calls whose finish_reason is length; leaving max_tokens unset works too (measured: 10,432 tokens, finish_reason=stop, complete file). No server flag can raise a client's explicit cap — --max-total-tokens only sizes the KV pool. On the input side this fork does drop upstream's --allow-auto-truncate, so an over-long prompt now fails with HTTP 400 instead of being cut silently. Details: docs/gb10-findings.md §9.

Why not 63 tok/s

Per speculative step this stack takes 72 ms vs ≈54 ms implied upstream (63.1 / 3.41), at equal acceptance (3.43 on code) and 99 % GPU busy. Ruled out by measurement: measurement method (TTFT 0.2 s), acceptance, draft MoE config, vision (A/B), library versions, hot-path source diffs between the PR head and 4ccff141, throttling and swap. Profile per step: 38 % BF16 GEMM (verify + draft lm_head), 21 % NVFP4 experts, 17 % FP8 dense, 10 % hyper-connection mix. The upstream author's private base image contained code no public SGLang has; that is the most likely difference. The hot-token map recovers most of the gap. Details and all tested levers: docs/gb10-findings.md.

Repo layout

scripts/       00-download, 02-build-trees, 03-probe-trees
bounded-ple/   01/01b image builds, Dockerfiles, 05 deploy, start/stop, PLE overlay, vendored rust/
build/         runtime overlays (loader, fp8, modelopt_quant, kda backend + stub), tree builders 06/07/08, probes
tokenmap/      hot-token map, build_map.py (byte-identical rebuild), corpora, gen_corpus.py
tests/         crash regression, decode bench, quality/NLL, GSM8K, golden, BF16 kernel microtest
results/       raw outputs of the 2026-09-17 runs incl. the clean-room validation
upstream-pr/   prepared (unsubmitted) SGLang patch for the fp8-KV extend crash
docs/          gb10-findings.md (this fork), runbook-spark2.de.md, upstream failure-history / verification / memory-budget

Licenses

• Recipe code and this fork's changes: MIT (LICENSE).
• bounded-ple/rust/ and SGLang-derived overlays in build/ and bounded-ple/overlay/: Apache-2.0 (LICENSES/Apache-2.0.txt).
• tokenmap/: token ids of the Qwen3.8 tokenizer and model outputs (Qwen Community License 1.0 notice in LICENSES/); corpus_de_wiki.txt.xz is Wikipedia text under CC BY-SA 4.0; corpus_local.txt.xz is SGLang source (Apache-2.0). See tokenmap/README.md.
• Weights are not included and stay under their own licenses. Qwen Community License 1.0: commercial Model-as-a-Service or AI Work Assistant use of the model needs a separate license from Qwen.

Measured on one GB10 in one day. Verify on your own hardware.