Qwen3.8-27B-heretic-ara-DSpark

This is a tuned DSpark speculator for https://huggingface.co/catplusplus/Qwen3.8-27B-heretic-ara-NVFP4-MTP. Since the model is abliterated and quantized, original MTP accuracy is reduced. This model is tuned to bring accuracy back up, to 3.46-4.10 mean acceptable length. I run it like this (uses some local scripts, just illustrating vllm arguments/environment variables):

dockless run -d --name seeker-inference
-e CUDA_HOME=/usr/local/cuda-13.0
-e C_INCLUDE_PATH=/usr/local/cuda-13.0/include
-e LIBRARY_PATH=/usr/lib/aarch64-linux-gnu/nvidia
-e FLASHINFER_NVCC=/usr/local/cuda-13.0/bin/nvcc
-e VLLM_USE_FLASHINFER_MOE_FP4=0
~/bin/unglitched_vllm
--trust-remote-code
--served-model-name Nikola
--gpu-memory-utilization 0.675
--port 9000
--enable-auto-tool-choice
--tool-call-parser qwen3_coder
--reasoning-parser qwen3
--mm-encoder-tp-mode data
--mm-processor-cache-type shm
--enable-prefix-caching
--max_num_batched_tokens 8192
--max-num-seqs 8
--cudagraph-capture-sizes 1 2 4 8
--mm-processor-kwargs '{"max_pixels": 3584000}'
--enable-chunked-prefill
--performance-mode interactivity
--attention-backend FLASHINFER
--async-scheduling
--moe-backend cutlass
--compilation-config '{"mode": 0, "cudagraph_mode": 0}'
--speculative-config '{"method": "dspark", "model": "/home/olegk/Nikola/models/Qwen3.8-27B-DSpark-tuned", "num_speculative_tokens": 7}'
--model "${@:-/home/olegk/Nikola/models/Qwen3.6-27B-uncensored-heretic-v2-NVFP4-MTP}"

Qwen3.8-27B-DSpark

A DSpark speculator for Qwen/Qwen3.8-27B-FP8. DSpark extends DFlash with target-model auxiliary features and a confidence head that dynamically chooses the number of draft tokens. The model was trained with SpecForge and is served with SGLang.

Model specifications

• Target model: Qwen/Qwen3.8-27B-FP8
• Checkpoint: epoch_2_step_4166
• Draft parameters: 1,359,284,737 (1.36B)
• Draft weight dtype: BF16
• Hidden size: 5,120
• Transformer layers: 5 full-attention layers
• Attention: GQA with 40 query heads and 8 key/value heads
• Target auxiliary feature layers: 4, 16, 28, 40, 52
• Confidence head: vanilla Markov head, rank 256
• DSpark block size: 7 draft tokens (verify width 8, including the target bonus token)
• Maximum position embeddings: 262,144

Acceptance length

Acceptance length is the mean number of tokens accepted per speculative verification step, including the target bonus token. Each workload value below is the unweighted mean of SGLang's per-request spec_accept_length values.

Setting: FP8 target and unquantized BF16 draft; DSpark block size 7; sampling temperature 0.6, top-k 20, top-p 0.95; thinking enabled; max_new_tokens=2048; seed 0; up to 128 prompts per workload, or the full set when smaller. The prompt construction and measurement loop follow z-lab/dflash's SGLang benchmark path.

The evaluation contains 1,164 requests. The mean across all requests is 3.39, and the macro mean across the 11 workload means is 3.35. Every response contained acceptance statistics, and recomputing completion_tokens / spec_verify_ct matched spec_accept_length exactly.

Serving with SGLang

Use an SGLang version containing DSPARK support (the results above used the pinned main-branch commit shown in the evaluation setting):

sglang serve \
  --trust-remote-code \
  --model-path Qwen/Qwen3.8-27B-FP8 \
  --tp-size 1 \
  --speculative-algorithm DSPARK \
  --speculative-draft-model-path RadixArk/Qwen3.8-27B-DSpark \
  --speculative-dspark-block-size 7 \
  --speculative-draft-model-quantization unquant \
  --mamba-scheduler-strategy extra_buffer \
  --attention-backend fa3