Qwen3.8-27B-Kearuga-DFlash2

🧙‍♂️⚡ Qwen3.8-27B-Kearuga-DFlash2

<p align="center"> <img src="https://huggingface.co/0xWhiteMage/Qwen3.8-27B-Kearuga/resolve/main/assets/header.png" alt="The White Mage — Qwen3.8-27B Kearuga on DGX Spark with SGLang and DFlash 2" width="100%"><br> <a href="https://huggingface.co/0xWhiteMage/Qwen3.8-27B-Kearuga-DFlash2/commits/main"><img src="https://img.shields.io/badge/Release-v1.0-blue.svg" alt="Drafter v1.0"></a> <a href="https://huggingface.co/0xWhiteMage/Qwen3.8-27B-Kearuga"><img src="https://img.shields.io/badge/%F0%9F%A4%97_Target-Qwen3.8_27B_Kearuga-blue.svg" alt="Target model"></a> <a href="https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2"><img src="https://img.shields.io/badge/%F0%9F%A4%97_Base-z--lab_DFlash_2-yellow.svg" alt="Base drafter"></a> <a href="https://github.com/0xWhiteMage/Qwen3.8-27B-Kearuga-SGLang-DGX-Spark-DFlash2"><img src="https://img.shields.io/badge/GitHub-Serving_repo-181717?logo=github" alt="Serving repo"></a> <a href="https://www.apache.org/licenses/LICENSE-2.0"><img src="https://img.shields.io/badge/License-Apache_2.0-blue.svg" alt="License: Apache 2.0"></a> <a href="https://x.com/0xWhiteMage" target="_blank"><img src="https://img.shields.io/badge/X-@0xWhiteMage-000000?logo=x&logoColor=white" alt="Follow on X"></a> <a href="https://ko-fi.com/0xwhitemage" target="_blank"><img src="https://img.shields.io/badge/Ko--fi-Donate-FF5E5B?logo=ko-fi&logoColor=white" alt="Donate on Ko-fi"></a> </p>

Kearuga's own DFlash 2 speculative drafter for 0xWhiteMage/Qwen3.8-27B-Kearuga — distilled on Kearuga's own outputs, quantized to NVFP4 (W4A4, ModelOpt layout) with activation scales calibrated on Kearuga's features, and served with a frequency-pruned 64K draft head.

---

⚡ Overview

On a single NVIDIA DGX Spark (GB10 / Grace-Blackwell, SM121), evaluated against the same Kearuga target served with the stock BF16 z-lab/Qwen3.8-27B-DFlash2 drafter at K = 10, this drafter delivers — paired, in one window, on one clock:

• 🎯 Lossless up to floating-point numerics: The target verifies every drafted token with its own untouched BF16 lm_head; a drafter can only change speed, not what the target outputs — up to bf16 rounding. Measured (2026-09-23, 40 prompts × 256 greedy tokens, thinking off): continuations from the speculative server and from the same target served without speculation diverge on 28/40 prompts, every first divergence at a top-2 logprob margin ≤ 0.25 (a bf16 near-tie); two runs of the speculative server itself diverge on 6/40 with the same margins — no divergence at a decided position. Fidelity-40 against the BF16 base is identical to the target served alone (mean KL 0.0165, JS 0.0034, top-1 agreement 40/40).
• 🗜️ 2.5× Smaller Footprint: 1.55 GB NVFP4 vs 3.85 GB BF16 — because the decode cycle on a GB10 is memory-bandwidth-bound, every byte eliminated from the weight stream directly saves execution time.
• 🏎️ Per-Domain Net Decode Speed (C1 Medians, 20 requests): Code 46.3 → 57.6 tok/s, Math 51.2 → 62.2 tok/s, Tool calls 80.8 → 114.6 tok/s, Prose 19.4 → 21.6 tok/s, Instruction-following 19.9 → 21.8 tok/s.
• 🛡️ Quality-200 Objective Gate: Passes 155 / 180 on the frozen Quality-200 suite (GSM8K 67/80 · HumanEval 38/40 · IFEval 34/40 · agentic coding 16/20), safely inside the pre-registered 154–160 band. (The target alone scores 157; all five candidate drafters landed in 154–159, reflecting the suite's near-tie noise floor rather than a drafter degradation).

[!NOTE] The absolute performance figures are measured on a GB10 with the target's full 262K context enabled and the owner's 2400 MHz SM clock lock active. Re-measuring with the lock lifted showed identical results (the GB10 natively executes this decode loop at 2.40–2.48 GHz; it is memory-bandwidth-bound, not compute-bound). These figures are not comparable to RTX 5090 / B200 setups — the GB10's 273 GB/s unified memory defines the physical ceiling, and this drafter maximizes bandwidth efficiency toward that ceiling.

---

📊 Paired Empirical Benchmark (Bake-Off, 2026-09-17)

All candidate configurations were evaluated in a single maintenance window, booted back-to-back on the exact same container image, target weights, flags, and hardware clock.

• Battery: 50 prompts (10 prompts × 5 domains) × 2 runs.
• Sampling: temperature 0 (greedy), thinking disabled, domain-calibrated max_tokens (code 700, math 600, etc.).
• Metric Definition: "Net decode tok/s" = (completion_tokens − 1) / (t_last_token − t_first_token) per request (TTFT excluded). Paired per-prompt deltas evaluated with bootstrap 95% confidence intervals.

1. Per-Domain Breakdown: This Drafter (K = 12, 64K Map) vs. Stock BF16 Baseline (K = 10)

• TTFT Impact: Median TTFT at C1 improved from 0.28–0.30 s down to 0.27–0.28 s (tool prompts accelerated from 0.78 s down to 0.62 s), as the first decode cycle executes a cheaper draft step.

2. Candidate Rankings (Pre-Registered Expected Value Rule)

Candidates were evaluated under the pre-registered ranking criterion: $$\text{EV} = 0.8 \cdot \sum w_d \cdot \Delta C1_d + 0.2 \cdot \sum w_d \cdot \Delta C4_d$$

• Workload Weights ($w_d$): Code: 0.30 · Tool: 0.20 · Math: 0.10 · Prose: 0.25 · Instruction-Following: 0.15 (frozen pre-bakeoff).
• Quality Gates: Fidelity-40 top-1 40/40, mean KL ≤ 0.0170, no C1 domain regression vs baseline, C4 aggregate ≥ −1%.

3. Exact Speculative Acceptance

Measured per request from the server's internal verify-call counter: $\frac{\text{completion_tokens} - 1}{\text{verify_calls}}$ (greedy, thinking off, K = 12, without draft-head map, $n = 5$ per domain):

Sampling and thinking-on (production resident: this drafter, K = 12, with the 64K draft-head map; 2026-09-17; $n = 2$ prompts per domain, sequential, verified free of concurrent traffic):

• Sampling costs little acceptance: −0.1 to −1.1 tokens/cycle vs greedy (prose unchanged at ≈ 2.5). The speculative-sampling verify path is exact, so sampled outputs remain the target's own distribution.

• Thinking-on is the real cost: with reasoning enabled, code acceptance falls from 7.5 to 3.1 tokens/cycle — the reasoning trace behaves like prose — so a thinking-on code request decodes at roughly prose speed (≈ 27 tok/s incl. TTFT on this probe vs ≈ 56–65 thinking off). This is the drafter's largest remaining headroom.

• Thinking on, paired vs the stock drafter (2026-09-22; same 50-prompt battery × 2 runs, reasoning_effort=xhigh, model-default sampling, every cell booted with --disable-flashinfer-autotune): this drafter 32.90 tok/s C1 / 102.33 C4 vs stock K = 10 30.61 / 94.00 (−7.0 % / −8.1 %) and stock K = 12 29.61 / 93.96 (−10.0 % / −8.2 %) — faster in all five domains (code 34.6 vs 31.9, math 49.3 vs 43.0, tool 75.2 vs 53.3, prose 23.6 vs 21.9, IFEval 37.8 vs 34.5 tok/s). Thinking-on acceptance at K = 12: code 3.15 · math 5.68 · tool 7.14 · prose 2.50 · IFEval 3.79 tokens/cycle (stock K = 12 accepts up to 0.5 more per cycle on four domains but its cycle costs more). Fidelity-40 40/40 in every cell.

• Reasoning effort matters more than the drafter for thinking-mode latency: the Kearuga chat template — like Qwen3.8's — defaults to reasoning_effort=xhigh. On the live resident (thinking on, model-default sampling, 2 prompts per domain, 4,096-token cap) both code prompts at xhigh spent the whole budget reasoning and never answered (135 s, 3.47 accepted/cycle), while medium answered in ≈ 90 s (5.05 accepted/cycle); on math medium cut wall time 53.7 → 20.4 s with acceptance 6.27 → 7.56. Recommendation for coding and agent clients: send chat_template_kwargs: {"reasoning_effort": "medium"} per request. (Full table in the serving repo README §1.)

• Impact of the 64K Draft-Head Map: Tokens outside the hot set represent only 0.44% of Kearuga's output tokens. In the paired bake-off table above, gauge-based acceptance deltas vs. the stock drafter stay within $\pm 0.1$ tokens on code, math, prose, and instruction-following at C1 (speed on tool calls increases +26% / +31%).

4. Fidelity-40 Verification (vs. Native Unquantized BF16)

• The target verifies every drafted proposal with its own untouched lm_head; the first-token distribution is mathematically independent of the drafter.
• Continuation identity shifts by $\pm 2\text{--}3$ tokens across boots or drafter profiles purely due to near-tie logits verified in different order; a single drafter reproduces its continuations 40/40 deterministically across reboots.

5. Against No Speculation (2026-09-23)

The same target, image and flags with every --speculative-* flag removed (single tenant, thinking off, temperature 0, the serving repo's 50-prompt battery):

Without a drafter the GB10 decodes this 24.85 GB target at a flat ≈ 10.3 tok/s per stream (weight-streaming bound). The drafter also costs KV capacity: without it the pool hits the 4 × 262,144-token cap with 21 GB unallocated; with it, 932,355 tokens. Method and raw numbers: serving stack v0.6.7, README §1.

---

🛠️ Build Methodology & Architecture

The drafter was developed across three dedicated optimization stages to maximize speculative decoding speed and minimize unified memory bus traffic on Grace-Blackwell GB10 hardware:

🎯 Stage 1 Technical Specifications: Distillation on Kearuga Outputs ("Run 2")

🗜️ Stage 2 Technical Specifications: NVFP4 Weights & Activation Calibration

⚡ Stage 3 Technical Specifications: 64K Draft Head (draft-vocab-v4-top65650.pt + SGLang Overlay)

---

🔬 Empirical Ablations: What Did Not Work

To save future researchers and engineers redundant exploration, the following approaches were evaluated and rejected based on empirical performance:

---

🚀 Serving & Deployment Guide

Validated on pinned Docker container lmsysorg/sglang@sha256:616a3e97f45191af975896cfa644279096cb31bd408a071c2e99ca7209c3cafe, NVIDIA Driver 580, Tensor Parallelism tp = 1.

1. One-Command Launch (Serving Repository)

The simplest deployment method is using the turnkey launcher from the GitHub serving repo:

git clone https://github.com/0xWhiteMage/Qwen3.8-27B-Kearuga-SGLang-DGX-Spark-DFlash2
cd Qwen3.8-27B-Kearuga-SGLang-DGX-Spark-DFlash2
cp .env.sample .env
./start-dflash2.sh

• Automatically verifies hashes against drafter/SHA256SUMS and launches with K=12 and the 64K draft head overlay.
• To run the baseline profile instead, set DRAFTER_PROFILE=stock in .env.

---

2. Standalone Docker Deployment Options

Option A: Plain SGLang (+7.9% C1 / +4.3% C4, Zero Code Modifications)

docker run --gpus all --shm-size 32g --ulimit memlock=-1:-1 --cap-add IPC_LOCK -p 8888:8888 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  lmsysorg/sglang@sha256:616a3e97f45191af975896cfa644279096cb31bd408a071c2e99ca7209c3cafe \
  python3 -m sglang.launch_server \
    --model-path 0xWhiteMage/Qwen3.8-27B-Kearuga \
    --served-model-name "Qwen3.8 27B Kearuga" --trust-remote-code \
    --speculative-algorithm DFLASH \
    --speculative-draft-model-path 0xWhiteMage/Qwen3.8-27B-Kearuga-DFlash2 \
    --speculative-draft-model-quantization modelopt_fp4 \
    --speculative-num-draft-tokens 12 \
    --speculative-draft-window-size 2048 \
    --attention-backend flashinfer \
    --kv-cache-dtype bf16 --mamba-ssm-dtype bfloat16 \
    --context-length 262144 --max-running-requests 4 --max-total-tokens 1048576 \
    --mem-fraction-static 0.85 \
    --host 0.0.0.0 --port 8888

Option B: With 64K Draft-Head Map & Overlay (+14.5% C1 / +10.8% C4)

Mount the 5 read-only overlay files over the container's sglang directory and provide the precomputed token map:

REPO=/path/to/Qwen3.8-27B-Kearuga-DFlash2      # Local clone of this model repository
OV=$REPO/sglang-overlay
docker run --gpus all --shm-size 32g --ulimit memlock=-1:-1 --cap-add IPC_LOCK -p 8888:8888 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v $REPO:$REPO:ro \
  -v $OV/sglang/srt/models/dflash.py:/sgl-workspace/sglang/python/sglang/srt/models/dflash.py:ro \
  -v $OV/sglang/srt/speculative/dflash_worker_v2.py:/sgl-workspace/sglang/python/sglang/srt/speculative/dflash_worker_v2.py:ro \
  -v $OV/sglang/srt/speculative/dflash_utils.py:/sgl-workspace/sglang/python/sglang/srt/speculative/dflash_utils.py:ro \
  -v $OV/sglang/srt/speculative/dflash_head_utils.py:/sgl-workspace/sglang/python/sglang/srt/speculative/dflash_head_utils.py:ro \
  -v $OV/sglang/srt/server_args.py:/sgl-workspace/sglang/python/sglang/srt/server_args.py:ro \
  lmsysorg/sglang@sha256:616a3e97f45191af975896cfa644279096cb31bd408a071c2e99ca7209c3cafe \
  python3 -m sglang.launch_server \
    --model-path 0xWhiteMage/Qwen3.8-27B-Kearuga \
    --served-model-name "Qwen3.8 27B Kearuga" --trust-remote-code \
    --speculative-algorithm DFLASH \
    --speculative-draft-model-path $REPO \
    --speculative-draft-model-quantization modelopt_fp4 \
    --speculative-num-draft-tokens 12 \
    --speculative-draft-window-size 2048 \
    --speculative-dflash-token-map $REPO/draft-vocab-v4-top65650.pt \
    --attention-backend flashinfer \
    --kv-cache-dtype bf16 --mamba-ssm-dtype bfloat16 \
    --context-length 262144 --max-running-requests 4 --max-total-tokens 1048576 \
    --mem-fraction-static 0.85 \
    --host 0.0.0.0 --port 8888

• Startup Verification: Successful initialization prints confirmation in logs:

  [kearuga-drafthead] loaded hot token map: rows=65650 …
  [kearuga-drafthead] draft head rows=65650 dtype=torch.bfloat16 bytes=672256000 fp8=False hot_map=True
  

---

3. Recommended Draft Block Size ($K$)

---

📦 Model Files & Assets

Note: Speculative drafters do not package a tokenizer; SGLang automatically leverages the target model's tokenizer.

---

⚠️ Operational Notes & Limitations

• 🎯 Tuned for Kearuga: Distilled and calibrated specifically on Kearuga's activations. While architecturally compatible with other Qwen3.8-27B variants, acceptance on third-party targets has not been benchmarked.
• 🖥️ Hardware Prerequisite: NVFP4 Tensor Core acceleration requires an NVIDIA Blackwell architecture GPU (e.g., GB10 / SM121) and SGLang with modelopt_fp4 support. Validated on tp = 1.
• 🎲 Greedy vs. Sampling: The paired bake-off statistics are greedy (temperature 0). Model-default sampling (T 1.0 · top-p 0.95 · top-k 20) was measured on the production resident at −0.1 to −1.1 tokens/cycle vs greedy (table in §3, $n = 2$ per domain); thinking-on code drops to ≈ 3.1 tokens/cycle. Neither has been through the full 50-prompt paired battery.
• 🔀 Near-Tie Continuation Divergence: Greedy continuations differ slightly from the stock drafter at near-tie logits (18/40 vs. 20/40 identical 32-token sequences vs. BF16). The target's underlying output distribution remains identical.
• 🧪 Offline evaluation caveat: the HF/z-lab reference DFlash loop is not lossless on this Gated-DeltaNet hybrid — transformers' Cache.crop leaves rejected drafts in the recurrent state (huggingface/transformers#49036, z-lab/dflash#172) and inflates offline acceptance (1.9× in our 100-prompt diagnostic). Measure acceptance on SGLang, or snapshot and restore the recurrent states around each verify block.
• 🔒 Digest-Pinned Overlay: The optional 64K overlay is verified against image lmsysorg/sglang@sha256:616a3e97…. Confirm base hashes via sglang-overlay/MANIFEST.json before applying to other builds.
• 📈 Benchmark Context: Performance was measured from a 50-prompt battery on one DGX Spark (GPU clock lock on or off made no measurable difference as the decode loop is memory-bandwidth bound); user prompts, sequence lengths, and cache states will vary.

---

🔗 Related Resources

• Target Model: 0xWhiteMage/Qwen3.8-27B-Kearuga — Tiered sensitivity hybrid-quantized production checkpoint
• Serving Stack & Benchmark Repo: 0xWhiteMage/Qwen3.8-27B-Kearuga-SGLang-DGX-Spark-DFlash2 — Production scripts and verification gates (v0.6.7; DRAFTER_PROFILE=kearuga pins this drafter release)
• Base Drafter & Architecture: z-lab/Qwen3.8-27B-DFlash2 — Block-diffusion speculative decoding
• Quantization Reference: maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal — Reference NVFP4 calibrated layout
• Author / Maintainer: The White Mage · @0xWhiteMage on X

---

🤝 Acknowledgements & Community Credits

This model builds upon key contributions across the open-source community:

Architecture, Base Weights & Benchmarks

• z-lab: For inventing DFlash block-diffusion speculative decoding and releasing the foundation z-lab/Qwen3.8-27B-DFlash2 weights.
• Inco / incoai/Qwen3.8-27B-DFlash2: For public per-task acceptance reference tables and technical analysis.
• SGLang (LMSYS / RadixArk): For the high-throughput inference engine, spec-v2 worker, and overlap scheduling architecture.
• NVIDIA TensorRT Model Optimizer: For the standard ModelOpt NVFP4 quantization format.

Drafter Quantization Research

• maurienne-ai: For pioneering calibrated NVFP4 DFlash 2 and demonstrating that activation calibration prevents RTN acceptance collapse.
• tcclaviger: For multi-precision GB10 benchmarks demonstrating the bandwidth flattening curve past FP8/FP4.
• lued, josch15366, syvai, gratex, YourHighnessLA, TechPrototyper, Mia-AiLab: For extensive open quantization recipes, exclusion profiles, and acceptance notes.

Drafter Training & Adaptive-K Methods

• DaoCloud: For the 1.95B-token speculative training corpus and SpecForge training methodology.
• alphakek: For insights on target-distance and drafter adaptation on specialized models.
• mrchuy: For bootstrapping drafter research.
• AEON-7: For dynamic batch-aware K tapering architectures.
• sxuff: For GB10 hardware profiling and FP4 drafter acceleration proofs.

Foundational Research Papers

• FR-Spec: Zhao et al., Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Sampling, ACL 2025 (arXiv:2502.14856, thunlp/FR-Spec).
• Spec-AUF (arXiv:2607.01893), D-PACE (arXiv:2605.18810), and Draft-OPD (arXiv:2605.29343).

---

📜 Citation

@misc{kearuga-dflash2-2026,
  title={Qwen3.8-27B-Kearuga-DFlash2: a Kearuga-distilled, Kearuga-calibrated NVFP4 DFlash 2 drafter with a frequency-pruned draft head for single-GPU DGX Spark serving},
  author={0xWhiteMage},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/0xWhiteMage/Qwen3.8-27B-Kearuga-DFlash2}
}

---

📄 License

Released under the Apache 2.0 License. Derived from z-lab/Qwen3.8-27B-DFlash2 (Apache-2.0). The underlying base target model Qwen/Qwen3.8-27B is subject to Alibaba Cloud's original license terms.