Downloads · 30 days
95
12% of all-time downloads
onewhosighs/Apathy-Qwen3.8-27B-DFlash-drafter-v2
Apathy-Qwen3.8-27B-DFlash-drafter-v2 is a machine learning model from onewhosighs. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
A DFlash-family draft model for speculative decoding with Qwen3.8-27B, trained against unsloth/Qwen3.8-27B-NVFP4. Published so the single-stream decode result below can be reproduced exactly.
Downloads · 30 days
95
12% of all-time downloads
All-time downloads
763
Public
Parameters
2.1B
4.3 GB on disk
Likes
2
Trending 1
Click a slice to open those files.
.safetensors4.3 GB · 100%
From the Hugging Face model README
A DFlash-family draft model for speculative decoding with Qwen3.8-27B,
trained against unsloth/Qwen3.8-27B-NVFP4. Published so the single-stream
decode result below can be reproduced exactly.
This is a draft model. It is not useful on its own — it proposes tokens that a target model verifies. On a DGX Spark (GB10), measured single-stream on a fixed code prompt, temperature 0, median of 5 deterministic repetitions:
| Configuration | tok/s |
|---|---|
Qwen3.8-27B + this drafter (--dflash-gamma 15) | 63.9 |
Qwen3.8-27B + public incoai/Qwen3.8-27B-DFlash2 (--dflash-gamma 7) | 43.9 |
| Qwen3.8-27B, no speculation | 13.9 |
That figure is workload-specific. Speculative decoding pays in proportion to how predictable the next tokens are. On the same setup, decode ranges from 63.9 tok/s on boilerplate-heavy code down to ~19 tok/s on open prose, median ~37 across mixed workloads. Plan against the low end for chat or prose.
| Architecture | DFlashDraftModel |
| Layers | 6 |
| Hidden size | 5120 |
| Vocab | 248320 |
block_size | 16 |
| Weights | BF16, 3.96 GiB, 69 tensors |
block_size sets the usable draft width: trained_drafts = block_size - 1, so
--dflash-gamma 15 is both optimal and maximal here. Asking for more is
refused by the loader. Deriving gamma from the drafter rather than accepting a
CLI default matters — the wrong value degrades acceptance silently.
With the Atlas inference engine on GB10:
docker run --rm --gpus all --ipc=host -p 8898:8898 \
-v /path/to/Qwen3.8-27B-NVFP4:/model:ro \
-v /path/to/this-drafter:/drafter:ro \
ghcr.io/theapathy/apathy-atlas:gb10 \
serve --model-from-path /model --port 8898 --bind 0.0.0.0 \
--kernel-target qwen3.8-27b \
--dflash --draft-model /drafter \
--dflash-gamma 15 --dflash-quantization nvfp4
The engine quantises the dense projections to NVFP4 at load time; the BF16 sources here are the input to that.
The tuning flags matter more than they look. The command above omits them for brevity and will run far below the numbers quoted here — speculation without a tuned verify path is actually slower than no speculation at all. The full 60-variable profile, the probe used for these measurements, and the caveats are at:
https://github.com/TheApathy/apathy-atlas/tree/perf/qwen38-gb10-dflash/bench/qwen38-gb10
Measured on DGX Spark / GB10 (sm_121f, unified memory, ~273 GB/s). The result
is bandwidth-bound and does not transfer unchanged to discrete GPUs.
Trained on completions generated by the target model itself. A known limitation: a substantial fraction of the training corpus was not on-policy, and it contained essentially no reasoning-span text while reasoning spans dominate what the drafter actually drafts at serve time. Measured serve-time acceptance is therefore below what training loss suggested. A successor addressing both is in progress.
Per-position acceptance is not flat — it dips around position 5 and rises through the tail, so acceptance is bimodal rather than uniformly mediocre.