Qwen3.8-Flash-Next

Qwen3.8-Flash-Next NVFP4 on 2× NVIDIA DGX Spark (GB10) with SGLang — full recipe

A production-verified recipe (2026-08-27) for serving RadixArk/Qwen3.8-Flash-Next-NVFP4 (125B MoE / 6B active + 51B n-gram PLE + 4B MTP, 135 GB) on two NVIDIA DGX Spark machines with TP=2 over RoCEv2 and SGLang. The RadixArk NVFP4 checkpoint is validated upstream only on GB300/B300; this recipe is the sm_121a (GB10) verification.

Measured performance (30-minute soak)

Hardware

• 2× NVIDIA DGX Spark (GB10, 128 GB unified LPDDR5x each, arm64).
• Direct 200G DAC link between the two, RoCEv2 (NCCL_IB_* in the manifest). No switch: a straight port-to-port cable with static IPs (10.0.0.1/10.0.0.2).
• The weights live on one node; the other reads them over NFS (optionally NFS-RDMA over the same link — we measured 5.5 GB/s).

Software

• Image: image/Dockerfile — SGLang with the qwen4_exp build (PR sgl-project/sglang#36497, branch qwen4-main-squashed@73a2552), modelopt_fp4 quantization, flashinfer with NVFP4 cutlass kernels for sm_121a, arm64. The branch requires cargo (Rust extensions) — the Dockerfile installs rustup.
• Deployment: k8s/qwen38-flash-next-nvfp4-sglang.yaml — two Deployments (head rank 0 + worker rank 1), hostNetwork (the ranks talk over the RoCE fabric IPs), and a ConfigMap with the full launch.sh. Every parameter is commented in the yaml itself with the reasoning behind it.

Critical parameters (the ones that cost blood)

Concurrency vs context: the real tuning axis (measured)

The KV cache of this model is almost free (12 KB/token/rank: only 12 of 48 layers carry KV -- the whole pool at fraction 0.78 was ~4.4 GB), while the mamba/GDN state costs ~0.5 GB per concurrent request (5 slots/request with NEXTN x ~0.1 GB/slot). The KV pool has no direct knob: it is the remainder fraction x 120 - weights(74) - mamba - graphs, so you steer it with --mem-fraction-static and --max-mamba-cache-size (always identical on both ranks). Three configurations measured on the same pair of Sparks:

The shipping config is 0.90 / 8 concurrent / ~1.37M-token KV pool (~5 full 262K sessions in parallel) with flashinfer autotune ON and ~7 GB of node headroom. It boots clean in ~8 min.

0.90 is the ceiling with autotune enabled. 0.94 does reach a ~1.7M KV pool but leaves the node with 2 GB, which hangs the flashinfer autotune (no buffer memory) for ~27 min and drags CUDA graph capture out to ~25 min; it only boots with --disable-flashinfer-autotune. Not worth the fragility.

Always check max_total_num_tokens in the boot log after touching either knob: one intermediate config left the KV pool at 67K tokens -- below the 262K context -- and long agent sessions silently stopped fitting.

Large-context concurrency on the 6cc/1M profile (prefix-cache-friendly corpus, so treat prefill figures as upper bounds):

A cold 72K-token single prompt (no prefix reuse) prefills in ~29 s (~2.5K tok/s).

Why not vLLM (yet)

There is an official vLLM recipe for this model. We evaluated it and stayed on SGLang for this hardware:

• The FP8 checkpoint is 172.8 GiB (~86.4 GiB/rank on TP2). Our NVFP4 checkpoint is ~74 GiB/rank -- and on 128 GB unified-memory boxes that 12 GiB difference is the whole safety margin. vLLM does not load the RadixArk experts-only NVFP4 layout, so with vLLM you are forced onto FP8.
• The PLE CPU-offload flag buys nothing on GB10: VLLM_PLE_CPU_OFFLOAD moves the 51 GB n-gram table "to host RAM" -- on unified memory that IS the same LPDDR. It only helps on discrete-GPU systems (GB300/H200).
• The recipe is validated single-node only (GB300, 8x H200, MI355X) and n-gram embedding does not support pipeline parallelism; two-Spark TP over RoCE with the PLE would be unexplored territory again.
• Independent validation is welcome though: the vLLM recipe also disables flashinfer autotune, uses 3 speculative tokens, and hits the same mamba-cache capacity wall -- three of our hard-won conclusions, reproduced.

If an official NVFP4 checkpoint that vLLM loads ever ships, it is worth re-evaluating: vLLM on these Sparks is otherwise well-trodden ground.

Running without Kubernetes (plain docker run)

run/head.sh and run/worker.sh reproduce the whole deployment with plain docker run on the two Sparks -- same image, same NCCL/RoCE environment, same engine flags, with the boot-order and GID-resolution notes inline.

The two former hot patches -- now baked into the image

The image built from the open branch ships two landmines that the ConfigMap's launch.sh patches before exec (search for QWEN_FA_STUB and QWEN_QSA_GATE in the yaml):

1. flash-attn FA2 with a broken ABI (unresolved symbol c10_cuda_check_implementation against the image's torch) whose __init__.py imports it unconditionally, killing the FA4 cute path too. The patch stubs the __init__ so only the cute path survives.
2. The FA4 cute path crashes anyway (an MLIRError about shape congruence in nvidia_cutlass_dsl while building the varlen kernel). The real way out: QSA's sparse decode has a third path, trtllm-gen via flashinfer, gated behind is_sm100_supported(), which excludes GB10 (sm_121). The patch opens the gate to sm120+ — with trtllm resolved, the backend never calls the broken varlen. (Related: sgl-project/sglang#36531.)

Both patches are now BAKED into the image as build layers (v0.2.0, see image/Dockerfile.baked-patches): the launch script is clean again. The definitive fix is still upstream (a flash-attn wheel built against the image's torch, and sglang accepting sm120 in the gate).

Memory go/no-go: the PLE

The n-gram PLE tables (51 GB) are FP8 on disk. The RadixArk model card says they dequantize to BF16 on load (which would not fit); measured on sm_121a: they stay FP8 — ~74 GiB of weights per rank, with plenty of pool left for the 262K context. Verify it on your boot: Load weight end ... avail mem should leave ~33–35 GB.

Operational gotchas

• Both ranks always together: if one restarts out of step, the other waits 601 s for the TCPStore rendezvous and dies. Restart both at once (coordinated Recreate).
• The first request after a boot takes ~10–60 s (warmup). It is not down.
• If the node serving the weights over NFS also does heavy I/O during boot, the other rank can get stuck in folio_wait_bit_common (a page-cache reclaim storm). Avoid large concurrent downloads.
• Metrics come with the sglang:* prefix (including sglang:spec_accept_length to watch the speculative decoding).
• Parsers verified in production: --reasoning-parser qwen3 and --tool-call-parser qwen3_coder (the template emits XML tool calls). Reasoning arrives in reasoning_content, tool calls come structured, and sglang#36537 (thinking + qwen3_coder looping) did not reproduce on this build — tested with thinking, tools, both combined, and streaming.

Sampling and thinking controls (from the official model card)

The model thinks by default, emitting <think>\n...</think>\n\n before the final answer. Recommended sampling, per mode:

presence_penalty can be raised between 0 and 2 to curb endless repetition, at the cost of occasional language mixing and a slight quality drop.

Thinking behaviour is controlled through the chat template (works through any OpenAI-compatible gateway via chat_template_kwargs):

• enable_thinking: false — direct answers, no <think> block.
• preserve_thinking (default true) — keeps the thinking blocks of ALL previous turns in the prompt. This is deliberate: it maximises decision consistency in agents and radix/KV-cache reuse. Set it to false to keep only the latest turn's thinking.
• reasoning_effort — supported levels: xhigh, medium, low. Note from the card, confirmed by our agent experience: in multi-turn agentic tasks a LOWER effort does not necessarily lower total task time — weaker analysis causes failures and retries that cost more than the faster turns save.

For agentic workloads the card recommends generous output budgets (reasoning up to 262,144 tokens and final response up to 131,072 where the serving stack splits the two). Size your gateway's max_output_tokens accordingly — our deployment uses a much smaller cap and it is a deliberate trade-off, not a model limit.

Extending context beyond 256K (YaRN)

We deliberately run at the native 262,144 (static YaRN penalises short contexts, and on 2× GB10 there is no memory for the 1M KV pool anyway). If you need more, the official recipe is to change rope_parameters inside text_config in config.json:

{
  "mrope_interleaved": true,
  "mrope_section": [11, 11, 10],
  "rope_type": "yarn",
  "rope_theta": 10000000,
  "partial_rotary_factor": 0.25,
  "factor": 4.0,
  "original_max_position_embeddings": 262144
}

Set factor to what you actually need (e.g. 2.0 for 512K): all open-source frameworks implement static YaRN, so an oversized factor degrades short prompts all the time.

Vision and video

Attention/GDN/vision run in BF16 in this checkpoint, so the multimodal path is intact. For hour-scale video the card recommends raising longest_edge in video_preprocessor_config.json to 469762048 (≈224k video tokens); see sglang PR #18467 for engine-side overrides. We have not benchmarked video on the 2× Spark setup — budget KV accordingly before trying.

Alternative: one Spark, GGUF

If you have a single DGX Spark (or want CPU offload of the 51 GB n-gram PLE), the Unsloth Dynamic GGUFs run through their llama.cpp PR ggml-org/llama.cpp#27742 (the n-gram embedding is exactly the kind of parameter mass that offloads well — it is indexed, not scanned). That path trades our TP=2 throughput and native FP4 experts for a one-box setup; this recipe is the full-quality, full-context two-node route.

Full troubleshooting

The catalogue of every failure mode we hit (with diagnosis and fix) is in docs/TROUBLESHOOTING.md.