Qwen3.8-Flash-Next-RadixArk-NVFP4-Hybrid-Sharp

[!NOTE] This checkpoint stores some layers in a mixed 4-bit/8-bit format that stock vLLM and SGLang cannot load. It requires the patched vLLM image from lancelind/qwen3.8-Flash-DGX served with VLLM_FP8_HYBRID=1. Built and validated on a single NVIDIA DGX Spark / GB10: the image reads the model's 44 GiB lookup table directly from disk instead of loading it into memory, which is what lets a ~176B-parameter model fit in 128 GB alongside enough working memory to actually serve requests. The model is multimodal, and image input works through this serving stack (verified with an image request against the running server); the measurements below are text-only workloads.

Qwen3.8-Flash-Next-RadixArk-NVFP4-Hybrid-Sharp

Stacks two independent contributors to inference efficiency on RadixArk's NVFP4 quantization of Qwen3.8-Flash-Next: hybrid fp8 side layers (faster decode per token) and the Sharp chat template (fewer tokens per answer). All weights other than the four converted side-layer shards are untouched — identical to RadixArk's release.

What's inside

• Base: RadixArk/Qwen3.8-Flash-Next-NVFP4 — NVFP4 W4A4 (NVIDIA Model Optimizer, routed experts only) of Qwen/Qwen3.8-Flash-Next. Note: RadixArk's model card describes their checkpoint as a private candidate release.
• Hybrid fp8 side layers: most of this model's weights are experts that only activate for some tokens, but ~15 GiB of "side" layers (attention projections and shared experts) are read in full for every generated token — so their size directly limits generation speed. This build stores those layers at 8-bit precision (blockwise fp8-e4m3, 128×128 blocks with per-block scales) instead of 16-bit, halving what has to be read per token. The expert weights stay exactly as RadixArk shipped them. 300 tensors across 4 shards were converted; the worst per-tensor relative error is 0.0354 (roundtrip-checked during conversion). Conversion tool by @Saren-Arterius (Apache-2.0); loading support by blazux/qwen3.8-Flash-DGX, where this format measured ~20% faster generation and 15–20% more room for context, with no change in benchmark quality scores.
• Serving image (required, not part of this repo): ghcr.io/lancelind/qwen38-flash-dgx, built from lancelind/qwen3.8-Flash-DGX (fork of blazux/qwen3.8-Flash-DGX). It layers eight patch groups on the official vllm/vllm-openai:qwen38-flash-next image (each inactive unless its flag is set):
  1. Serve the model's 44 GiB lookup table straight from disk (VLLM_PLE_MMAP=1) — what makes the checkpoint fit in 128 GB
  2. Two GPU-kernel fixes for this hardware, by @Saren-Arterius — without them, 36 of the model's layers silently ran in a slow fallback mode, and a race condition (fla#953) could corrupt results
  3. A fix for a crash-causing race when the server copies internal state between requests (vllm#50729 by @AndreasKaratzas), hardened further by @Saren-Arterius
  4. A fix that makes response caching safe — without it, a cache hit silently produced wrong answers
  5. Deterministic decoding (VLLM_QSA_EXACT_TOPK=1) — the stock attention kernel on this GPU gives different results run to run and can drop the tokens it should have picked (vllm#51782)
  6. The loader for this checkpoint's mixed 4-bit/8-bit format (VLLM_FP8_HYBRID=1)
  7. A fix that makes tool_choice: "required" and named tool choice actually enforced on the chat path (verified 24/24 on-box)
  8. GPU table gather (VLLM_PLE_GPU_GATHER=1) — the disk-served lookup table is read by a CUDA kernel directly through the GB10's coherent address translation, inside the CUDA graphs, with the page cache warmed ahead of every read. This is what the measured serving numbers below come from: up to 2× total generation throughput at concurrency.
• Sharp chat template from peculiar-ragdoll/Qwen-Sharp-Chat-Templates (commit 3dc34df, v22.1 line), embedded in place of the original in both chat_template.jinja and tokenizer_config.json: more information per token by cutting filler without sacrificing correctness.

Measurements

Chat-template comparison, run on the same hardware against the standard (non-hybrid) build of this checkpoint's base, with deterministic settings and a 4-prompt test set: Sharp produced 18% fewer output tokens than the stock template (1714 vs 2089) with no wrong answers introduced — and on the longest prompt, Sharp finished a complete answer in 762 tokens where the stock template hit the 1200-token limit mid-answer. Sharp adds ~113 tokens of fixed instructions to each request's input, but that prefix is identical every time, so the server's response cache absorbs it after the first request.

Serving performance (this build, GPU table gather)

Measured 2026-08-30/31 UTC on one DGX Spark, this exact checkpoint, vLLM's own benchmark tool (2,048-token prompts, 256-token replies, 4×C requests per concurrency level). The only change between the two columns is the serving recipe: the current one gathers the 44 GiB n-gram table with a CUDA kernel inside the CUDA graphs (speculative decoding off, --gpu-memory-utilization 0.75) vs the previous CPU-threadpool gather (MTP=2, 0.85).

Single-user time to first token: mean 1942 → 1353 ms, P99 4003 → 1633 ms. The lower gpu-memory-utilization returns memory to the OS page cache serving the table, which removes cold-gather stragglers from the tail.

Why the recipe changed: speculative decoding (MTP) only pays off for a single stream — at concurrency, verifying draft tokens re-reads the model's per-token weights each step and costs more than it saves. Turning it off and freeing GPU-reserved memory for the page cache doubled total throughput at 10 concurrent users and cut tail latency ~2.5×.

Serving

Docker (any GB10 setup)

With VLLM_PLE_GPU_GATHER=1 the disk-served table lookup runs as a CUDA kernel, so it is safe inside CUDA graphs and vllm::ple_mmap_lookup must not appear in splitting_ops — the rest of the --compilation-config below is still required. (If you disable the GPU gather and run the CPU lookup instead, you must add "vllm::ple_mmap_lookup" back to splitting_ops, or graph replay skips the lookup and the server crashes or returns garbage.)

Download the checkpoint first, then serve with HF_HUB_OFFLINE=1. The offline flag is required: it makes vLLM resolve the repo id to the local snapshot path, and the disk-served table lookup only works with that local path — without it the boot fails with "PLE mmap: table path ... is not a local directory". --network host matches the validated setup (Docker's default bridge network needs working DNS that some DGX Spark installs lack; with host networking the server is on port 8000 directly).

hf download travelinlance/Qwen3.8-Flash-Next-RadixArk-NVFP4-Hybrid-Sharp

docker run -d --gpus all --ipc=host --shm-size 16g --network host \
  -v ~/.cache/huggingface:/hf -e HF_HOME=/hf -e HF_HUB_OFFLINE=1 \
  -e VLLM_PLE_MMAP=1 -e VLLM_PLE_GPU_GATHER=1 -e VLLM_PLE_DECODE_WARM=1 \
  -e VLLM_FP8_HYBRID=1 -e VLLM_USE_DEEP_GEMM=0 \
  -e VLLM_QSA_EXACT_TOPK=1 -e VLLM_USE_FLASHINFER_SAMPLER=1 \
  --entrypoint "" ghcr.io/lancelind/qwen38-flash-dgx:latest \
  vllm serve travelinlance/Qwen3.8-Flash-Next-RadixArk-NVFP4-Hybrid-Sharp \
    --served-model-name qwen3.8-flash-next \
    --host 0.0.0.0 --port 8000 --load-format safetensors \
    --max-model-len 262144 --max-num-seqs 8 --gpu-memory-utilization 0.75 \
    --enable-prefix-caching --enable-chunked-prefill --max-num-batched-tokens 8192 \
    --compilation-config '{"cudagraph_mode":"PIECEWISE","splitting_ops":["vllm::unified_attention_with_output","vllm::unified_mla_attention_with_output","vllm::mamba_mixer2","vllm::mamba_mixer","vllm::short_conv","vllm::qwen3_8_flash_next_ple_short_conv","vllm::qwen3_8_flash_next_qsa_with_output","vllm::linear_attention","vllm::qwen_gdn_attention_core","vllm::qwen_gdn_attention_core_fused_norm_packed","vllm::sparse_attn_indexer"]}' \
    --no-enable-flashinfer-autotune --kv-cache-dtype auto \
    --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3

Keep --gpu-memory-utilization at 0.75: the freed memory goes to the OS page cache serving the 44 GiB table, and raising it back to 0.85 costs throughput at concurrency and tail latency. Speculative decoding (MTP) is off on purpose — see the performance section above.

See lancelind/qwen3.8-Flash-DGX for tuning notes and the recipe this mirrors.

sparkrun (DGX Spark)

sparkrun registry add https://github.com/lancelind/qwen3.8-Flash-DGX
sparkrun run @qwen38-flash-dgx/qwen38-flash-hybrid-sharp-v2-vllm --solo

Attribution

This build stands on a lot of other people's work:

• Qwen — the Qwen3.8-Flash-Next model (Qwen Community License 1.0).
• RadixArk — the NVFP4 quantization this checkpoint derives from, produced with NVIDIA Model Optimizer.
• @blazux — qwen3.8-Flash-DGX: the PLE mmap design, prefix-caching block_size root cause and fix, exact top-k path, hybrid dispatch for ModelOpt-NVFP4, and the GB10 serving recipe this checkpoint requires.
• @Saren-Arterius — fp8 conversion tool (Apache-2.0), the GB10 FLA shared-memory and warp fixes, the faster PLE gather, the mamba state-copy bounds guard, and the original int4+fp8 hybrid dispatch the ModelOpt port descends from.
• @AndreasKaratzas — vLLM #50729, the mamba state-copy race fix carried in the serving image.
• @k3dani — diagnosis and upstream report of the GB10 non-deterministic sparse-attention top-k (blazux#3, vLLM #51782) that the exact top-k path fixes.
• @jschmied — independent reproduction on a DGX Spark, the native-offload fixes, and the concurrency measurements (write-up).
• vLLM — the serving engine and the vllm/vllm-openai:qwen38-flash-next base image (PR #53896) everything here runs on.
• peculiar-ragdoll — the Sharp chat templates, building on froggeric's template line (the embedded template versions itself qwen3.8-froggeric-v22.1).
• travelinlance — fp8-hybrid conversion of this checkpoint, Sharp template integration and on-Spark template A/B, the GPU table gather (CUDA-graph-capturable ATS gather with page-cache warming, replacing the CPU lookup) and the measured serving recipe built on it, and the sparkrun packaging.

License

Qwen Community License 1.0 — the license file is included in this repo, as its terms require for all copies. Redistribution and derivatives are permitted. Two conditions to know about: products over 100M monthly active users or US$20M monthly revenue must display the model name prominently, and "Model as a Service" or "AI Work Assistant" businesses need a separate license from Qwen for commercial use.