Downloads · 30 days
340
14% of all-time downloads
Koopah/Qwen3.6-35B-A3B-NVFP4-DSPARK-v2-GGUF
Qwen3.6-35B-A3B-NVFP4-DSPARK-v2-GGUF is a machine learning model from Koopah. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
GGUF conversions of Koopah/Qwen3.6-35B-A3B-NVFP4-DSPARK-v2 for llama.cpp speculative decoding. See the parent card for the full model description, training status, acceptance evaluations and the vLLM/sglang serving nu…
Downloads · 30 days
340
14% of all-time downloads
All-time downloads
2.3K
Public
Repo size
26.5 GB
Likes
3
Public
Click a slice to open those files.
.gguf26.5 GB · 100%
From the Hugging Face model README
GGUF conversions of Koopah/Qwen3.6-35B-A3B-NVFP4-DSPARK-v2 for llama.cpp speculative decoding. See the parent card for the full model description, training status, acceptance evaluations and the vLLM/sglang serving numbers — this repo only carries the llama.cpp artifacts and deployment instructions.
| file | dtype | size | note |
|---|---|---|---|
Qwen3.6-35B-A3B-NVFP4.gguf | NVFP4 | 23.8 GB | the target model — native NVFP4 conversion of unsloth/Qwen3.6-35B-A3B-NVFP4 at revision 612d523c58 (310 NVFP4 tensors with their scale sidecars, not re-quantized); loads on the performance branch. ⚠️ upstream later replaced its weights with a different re-export that fails our quality testing — the validated safetensors are frozen at Koopah/Qwen3.6-35B-A3B-NVFP4 |
Qwen3.6-35B-A3B-NVFP4-DSPARK-v2-BF16.gguf | BF16 | 1.04 GB | draft, recommended — matches the vLLM deployment dtype |
Qwen3.6-35B-A3B-NVFP4-DSPARK-v2-F16.gguf | F16 | 1.04 GB | draft, identical accuracy in practice |
Qwen3.6-35B-A3B-NVFP4-DSPARK-v2-Q8_0.gguf | Q8_0 | 0.56 GB | draft, smallest; ~equal acceptance, marginally different speed profile |
Target + draft together are everything the deployment command below needs — the served pair is bit-for-bit what every benchmark on this card measured.
Recommended: the performance branch —
KoopahTManiac/speed.llama.cpp
(default branch). Upstream llama.cpp implements DSPARK
(--spec-type draft-dspark) and serves this draft correctly; the
performance branch is upstream plus a series of speed commits for this
serving path — a fused in-graph truncation-sampler op, MoE finalize /
weight-load fusions, batched elementwise launches, and host-path fixes —
plus exact ratio acceptance (min(1, p/q) with exact residual
sampling) for sampled drafting. Output quality is identical to running
the target model alone, at any temperature (verified distributionally at
N=3200 completions/config). The benchmark numbers below are from the
performance branch.
# build (CUDA)
git clone https://github.com/KoopahTManiac/speed.llama.cpp
cd speed.llama.cpp && cmake -B build -DGGML_CUDA=ON && cmake --build build -j
# serve: target = the NVFP4 GGUF of Qwen3.6-35B-A3B, draft = this repo
./build/bin/llama-server \
-m Qwen3.6-35B-A3B-NVFP4.gguf \
-md Qwen3.6-35B-A3B-NVFP4-DSPARK-v2-BF16.gguf \
--spec-type draft-dspark --spec-draft-n-max 8 \
-ngl 99 -fa on -c 8192
--spec-draft-n-max is the drafted block length k (8 = fastest
single-user and the trained block size — larger values are clamped to
8; 4–6 shift the throughput/latency trade at batch, same as the
k-family curves on the parent card).
Optional — confidence-scheduled drafting (--spec-sched): the
performance branch implements the DSpark paper's confidence scheduler
using this draft's trained confidence head — hardware cost curve
profiled at startup, sequential temperature scaling self-calibrated
from live traffic and cached per model, causal expected-throughput
draft admission. Zero configuration; add --spec-sched-adapt for
adaptive depth + skip-drafting. On fast GPUs at short/mid context it
measures as parity with fixed-depth drafting — use it where fixed-depth
speculation loses to plain decoding (very long contexts, hard/creative
content, weaker GPUs), and leave it off for batched serving
(llama-server's host loop, not verify compute, is the batch
ceiling).
Same protocol as the parent card's engine sections (16 concatenated
GSM8K questions, output forced to 1024 tokens, temperature 0.7, 3-round
average, RTX PRO 6000 Blackwell 96 GB), llama-server, BF16 drafter,
dedicated single-user config (-np 1):
| config | solo (bs1) | vs vLLM solo (580) |
|---|---|---|
| DSpark k=8 | 506 tok/s (accept 5.88) | 87% |
| DSpark k=8, upstream llama.cpp base | 278 | 48% |
| no speculation | 201 | 109% |
(Upstream measured under the identical protocol and config — it gains
nothing from -np 1; its per-round overhead dominates regardless of
slot count.)
llama.cpp wins plain single-stream decoding outright and reaches ~87% of vLLM with speculation; acceptance matches or beats vLLM on this workload (6.92 vs 6.84 mean accepted length), so the residual gap is per-round host/launch overhead, not draft quality.
Per-category verification on
nvidia/SPEED-Bench
(qualitative split, all samples, temp 0, OSL 1024, single user) via the
in-tree speed-bench harness — baseline llama-server vs DSpark k=8
with Qwen3.6-35B-A3B-NVFP4-DSPARK-v2-BF16.gguf:
| category | base t/s | spec t/s | decode speedup | latency speedup | accept rate |
|---|---|---|---|---|---|
| coding | 201.5 | 442.4 | 2.20x | 2.07x | 0.51 |
| humanities | 201.6 | 348.0 | 1.73x | 1.82x | 0.36 |
| math | 201.6 | 403.4 | 2.00x | 2.13x | 0.47 |
| qa | 201.5 | 354.5 | 1.76x | 1.65x | 0.37 |
| rag | 201.6 | 432.7 | 2.15x | 1.99x | 0.49 |
| reasoning | 201.5 | 360.6 | 1.79x | 1.93x | 0.39 |
| stem | 201.4 | 354.0 | 1.76x | 1.91x | 0.38 |
| writing | 201.3 | 317.8 | 1.58x | 1.49x | 0.33 |
| multilingual | 201.3 | 445.0 | 2.21x | 2.05x | 0.52 |
| summarization | 200.9 | 398.4 | 1.98x | 1.77x | 0.45 |
| roleplay | 200.5 | 349.7 | 1.74x | 1.76x | 0.37 |
| overall | 201.3 | 382.4 | 1.90x | 1.85x | 0.41 |
Known limitation — batch serving: llama-server's per-slot host loop
(single-threaded sampling + stream writes) currently caps served batch
throughput near 700 tok/s aggregate even though the raw engine scales to
~1,900 tok/s at bs64 (llama-batched-bench). Use vLLM or sglang for
at-capacity serving; llama.cpp is the single-user/local option. At high
slot counts also pass --cache-ram 0 (host prompt-cache eviction can
livelock with this model's large per-slot state) and size -ub ≥
slots × 8 (the draft block is non-causal).