gb10-dgx-spark-llm-optimizations

NVIDIA GB10 / DGX Spark — LLM performance engineering

Optimized NVFP4 checkpoints and reproducible recipes for serving Qwen3.6-class and DeepSeek-V4-class models on NVIDIA GB10 (DGX Spark, sm_121, 119.6 GiB unified memory) with vLLM.

Everything here was measured on real hardware — a 2-node GB10 cluster running production agentic workloads (tool-calling agents, coding CLIs, chat). Numbers are single-stream decode with 512 output tokens unless stated, validated against vLLM's Prometheus counters.

---

📦 Models

---

⚡ Measured speeds (GB10, single stream)

35B MoE tooling model

Prefill @131K: 47.8 s (was 518 s → 10.8× faster). TTFT cached @131K: 1.9 s.

27B dense models

The optimized uncensored 27B decodes faster than the stock dense 27B — the entire gap was the lm_head.

---

🔧 The recipe (what actually moves the needle on GB10)

1. Quantize the lm_head. Community fine-tunes almost always leave it in BF16 (1–2.5 GB over a 248,320-token vocab). It is read every decode step, so on bandwidth-bound hardware it dominates. Quantizing it to 4-bit was worth +47% on the 27B and was the last piece to beat the dense model. NVIDIA's own NVFP4 checkpoints already ship it 4-bit — that is why they decode fast. Tip: quantize the lm_head standalone (load only that tensor, ~21 GB peak) — a full-model re-quantization OOMs on GB10's unified memory (the pack step alone transiently wants 18.95 GB).
2. Graft the MTP head when a fine-tune dropped it. Many fine-tunes declare MTP in config.json but publish no mtp.* tensors, silently disabling speculative decoding. Copying the 19 mtp.* tensors from an architecture-identical base restores it: ~100% acceptance at K=3, roughly 2.8× decode. Draft and verify share the target's weights, so fine-tune behaviour is preserved.
3. MTP K=3, not K=4. At K=4 the accept rate at position 4 falls below breakeven → net regression.
4. KV cache fp8, never nvfp4. See hardware limits below.
5. FlashInfer attention + Marlin NVFP4 GEMM (VLLM_NVFP4_GEMM_BACKEND=marlin, VLLM_MARLIN_USE_ATOMIC_ADD=1), --enable-prefix-caching, --enable-chunked-prefill.
6. Sampling matters for tool use. Qwen forbids greedy decoding: temperature 0 does not make tool-calling deterministic-and-safe, it causes degradation and infinite repetition loops. Use temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5 (the penalty is the recommendation for quantized checkpoints).

---

🧪 Tests & benchmarks we ran

• Speed harness: exact-length prompts built via /tokenize, cached + uncached passes, decode/prefill separated, every run validated against vLLM's Prometheus counters, max-of-clean-runs methodology, SHA-256 recorded per result ledger. Contexts: 1K / 8K / 16K / 32K / 131K (and a 245K pass).
• Long-context correctness: needle-in-a-haystack at 8K and 32K — HIT on the optimized checkpoints (quantizing the lm_head did not break retrieval).
• Tooling battery: 370 deterministic cases (tool chains, structured output, schema discipline, NL→tool+args inference). Ornith 346/370 after the lm_head W4 (343/370 before → within noise). Strong on chains/structured/schema discipline; weakest at NL inference of tool+args (76%).
• Agentic client tests: OpenClaw exact-one 20/20 (~7 s/turn); top-5 27/30.
• Speculative-decoding acceptance: measured per position; ~100% at K=3, sub-breakeven at position 4.
• Uncensored checks: refusal battery (compliance preserved after lm_head re-quant), coherence probes, upstream Heretic abliteration KL ≈ 0.0021 (minimal-damage abliteration preferred over heavier "uncensored" tunes, which measured worse on quality).
• Concurrency: continuous batching sweeps c1→c64 (the 35B MoE serves ~25k tok/s aggregate across 2 nodes; the dense 27B ~6k tok/s).
• Vision integration (DeepSeek-V4-Flash-0731): speculative-decoding acceptance measured from Prometheus counters before/after each wrapper fix (0.0124 → 0.600 in-window), KV pool sizing at 384K, isolated decode runs, and an image-comprehension sanity check. See the vision write-up.
• Base-model survey: 74 Qwen3.6-35B-A3B checkpoints on the Hub assessed programmatically (arch, quant format, real presence of mtp.* tensors in the safetensors index, lm_head precision, license, published benchmarks) to decide whether a better tooling base existed. Conclusion: none clearly better than the RL-post-trained Ornith for agentic tool use; plain re-quants of the same base are equivalent, and uncensored/roleplay/reasoning-distill variants are worse on the tooling axis.

---

👁️ Vision on a text-only model (DeepSeek-V4-Flash-0731)

DeepSeek-V4-Flash-0731 ships text-only. Bolting on a community vision stack (FlyCockpit's DeepEncoderV2 tower + adapter, served via a vLLM plugin) works on 2× GB10 — but the wrapper class breaks speculative decoding in two ways, one of which is completely silent:

• EAGLE3: isinstance(model, SupportsEagle3) runs against the top-level class, now the wrapper. One missing Protocol member aborts rank 1 — and in multi-node TP the traceback lands in the worker's log while the head just says Failed core proc(s): {}.
• lm_head: dspark/utils.py reads it with getattr() off the top-level object. The checkpoint sets has_own_lm_head=False, so the drafter must borrow the target's head. Without the attribute the sharing is skipped and the drafter keeps an unloaded head: acceptance 1.2%, ~10 tok/s, zero errors anywhere.

Both fixed by delegating to the backbone. Full write-up, measurements and the patch: deepseek-v4-flash-vision-gb10.md · dsv4_vision_wrapper_speculative_patch.py

The tower costs 1.8 GiB of KV permanently; it does not run on text-only requests (_execute_mm_encoder early-exits). Served at 384K context, TP=2, DSpark k=5.

---

⚖️ Unsloth Dynamic GGUFs vs this stack (weights + vision)

Unsloth's August 2026 GGUFs (Qwen3.8-27B Dynamic v3.0, DeepSeek-V4-Flash-0731 Dynamic 2.0 + a DSpark drafter GGUF) raise the obvious question: should the "lossless" UD-Q8_K_XL replace the native checkpoint served over vLLM TP=2? No — ~97% of DeepSeek-V4-Flash-0731 (all routed experts, 296.35B of 304.18B params) is natively MXFP4 and Unsloth repacks those bits identically, so Q8 has no quality headroom over the official safetensors this stack already serves; and llama.cpp's multi-node path (rpc-server, plain TCP) would give up the nvfp4_ds_mla KV pool, prefix caching, DSpark and the RoCE fabric. Where the GGUFs do win: a single-node fallback quant, and one-file mmproj vision — though for video, vLLM's native multimodal path has no llama.cpp equivalent. Full comparison with measured numbers: unsloth-gguf-vs-vllm-gb10.md

---

🚧 Hardware & ecosystem findings (GB10-specific)

• --kv-cache-dtype nvfp4 is impossible on GB10. FlashInfer requires sm100f (Blackwell datacenter); GB10 reports sm_121 and the engine aborts at startup. fp8 is the floor and the ceiling for KV on this hardware. Halving the KV pool via 4-bit KV is simply not available here.
• Unified memory is not free real estate. CPU and GPU share the 119.6 GiB, so a "small" transient allocation (e.g. an 18.95 GB pack buffer) OOMs the whole box. Prefer surgical, single-tensor operations over full-model pipelines.
• A BF16 lm_head is the #1 hidden decode tax on quantized community checkpoints.
• Fine-tunes silently lose MTP. Always check the safetensors index for mtp.* before assuming speculative decoding works — config.json declaring MTP is not evidence.
• Wrapping a model class can silently disable speculative decoding. vLLM's spec-decode paths probe the top-level model object with isinstance() and getattr(). A wrapper that delegates the forward pass correctly can still fail those probes — and getattr-based capability sharing fails open, so the engine degrades to ~1% draft acceptance without logging anything. Assert the interfaces at build time, not at deploy time.
• Broken chat templates cause "model" bugs. We traced constant tool-call loops in an agentic CLI to an unfinished edit in a fine-tune's chat_template.jinja (the conditional that strips stale <think> from history was deleted, while its last_query_index computation was left behind), verified by hashing it against the byte-identical canonical templates from Qwen/ and nvidia/. Details in the Ornith repo card.

---

Recipes, flags and env vars are documented in each model card so they can be reproduced end-to-end. Questions and corrections are welcome via the repo discussions.