Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4

Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4

This is an NVFP4 (4-bit floating point) quantization of nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451, a bf16 nuslerp mergekit merge of two Qwen3.6-27B variants, produced with NVIDIA TensorRT Model Optimizer 0.45.0 so it fits and runs on a 128 GB DGX Spark through vLLM's Blackwell FP4 path. No training or fine-tuning was involved: this is a numeric format conversion of somebody else's finished checkpoint. It is still a lossy one, and no quality evaluation of the 4-bit result against the bf16 source was run here (see Limitations). The two things this card is worth reading for are mtp.fc.weight -- the tensor vLLM's multi-token-prediction speculative decoding needs, which is absent from checkpoints pulled before 2026-07-21 and whose absence fails silently -- and the measurement of what MTP and DFlash speculative decoding are actually worth, which turns out to depend far more on the workload than on the concurrency.

---

Read this first: the missing mtp.fc.weight

If you are using any Qwen3.6-27B checkpoint with MTP speculative decoding, check for mtp.fc.weight before you trust a single throughput number.

The upstream base model was re-uploaded on 2026-07-21T18:30:00Z (commit 646f37367f, "Add files using upload-large-folder tool"). That upload added one tensor: mtp.fc.weight, shape [5120, 10240], BF16, 104,857,600 bytes: the fusion projection that concatenates the embedding and hidden streams before the MTP head's single decoder layer. The index at an earlier revision (5ae530c3ab) has 1198 tensors and no mtp.fc.weight; the index at main has 1199 and does. Any checkpoint (upstream or a derivative quantization) pulled before that date has mtp.layers.0.* and mtp.norm but no mtp.fc.

Why this failure is silent. vLLM only enables its strict weight-initialisation check for non-quantized models. From vllm/model_executor/model_loader/default_loader.py:

# We only enable strict check for non-quantized models
# that have loaded weights tracking by default.
default_enable_weights_track = (
    model_config.quantization is None and loaded_weights is not None
)

On a quantized checkpoint that check is off, so the missing parameter is never reported. ColumnParallelLinear allocates it, nothing loads into it, and it keeps whatever the allocation left there. The MTP head then drafts from a random projection: no exception, no warning, no error in the log. Acceptance collapses, every drafted token is rejected and re-verified, and you pay the drafting cost for nothing. This card does not contain a timed measurement of that specific broken state, but it does show what very low acceptance costs: the DFlash rows below run at 0.063 acceptance on prose and land 77.0% below the no-speculation baseline at N=64. The only symptom is a number you were not measuring.

vLLM's own source treats this tensor as a known special case (vllm/model_executor/models/qwen3_5_mtp.py):

# Workaround: mtp.fc is stored as BF16 in NVFP4 checkpoints but is
# missing from hf_quant_config.json exclude_modules. Force unquantized.
# Ref: https://github.com/vllm-project/vllm/pull/38650
# Ref: https://github.com/NVIDIA/Model-Optimizer/pull/1124

Check your own copy

import json
from huggingface_hub import hf_hub_download

repo = "PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4"  # or your own
idx = json.load(open(hf_hub_download(repo, "model.safetensors.index.json")))
wm = idx["weight_map"]

print("tensors:", len(wm))
print("mtp.fc.weight:", "mtp.fc.weight" in wm)
print("mtp.*:", sorted(k for k in wm if k.startswith("mtp.")))

mtp.fc.weight: True is what you need before enabling MTP. If it prints False, do not use --speculative-config '{"method":"mtp",...}' with that checkpoint. You will get a silent slowdown, not an error.

Which build of this model has it

Two artifacts come out of this quantization, and only one of them is safe to run MTP on:

Run the snippet above against whichever one you pulled rather than trusting this table. The MTP throughput rows further down were measured on the -MTP build; the no-speculation and DFlash rows were measured on the plain -NVFP4 build, which is immaterial to those two because neither touches the MTP head.

For a local directory, read model.safetensors.index.json directly. If you want vLLM to fail loudly on any missing tensor rather than trusting the index, force the strict check back on at serve time:

--model-loader-extra-config '{"enable_weights_track": true}'

(enable_weights_track is an accepted key of --model-loader-extra-config in vLLM 0.25.x; setting it true overrides the quantized-model default and raises ValueError: Following weights were not initialized from checkpoint: {...}.)

---

Serving

All three modes below were run on the same box with the same flags; only --speculative-config differs.

Common flags used for every measurement in this card:

--kv-cache-dtype fp8
--attention-backend flashinfer
--max-model-len 262144
--gpu-memory-utilization 0.72
--max-num-seqs 64
--reasoning-parser qwen3
--trust-remote-code
--enable-auto-tool-choice
--tool-call-parser qwen3_xml

Every measured run also carried --served-model-name spec-test and a local request-logging middleware (--middleware vllm_streams.StreamTracker) that is not part of vLLM and is not needed to reproduce the serving behaviour; neither is listed above because neither affects how the model is served.

vLLM version under test: 0.25.2.dev0+g752a3a504.d20260721, in the community Grace-Blackwell image ghcr.io/timothystewart6/vllm-gb10:v0.25.1-gb10.2.

1. No speculative decoding

vllm serve PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4 \
  --kv-cache-dtype fp8 --attention-backend flashinfer \
  --max-model-len 262144 --gpu-memory-utilization 0.72 --max-num-seqs 64 \
  --reasoning-parser qwen3 --trust-remote-code \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml

The quantization format is read from the checkpoint's own config; no --quantization flag is needed.

2. MTP (multi-token prediction, in-checkpoint draft head)

Add:

--speculative-config '{"method":"mtp","num_speculative_tokens":2}'

This uses the MTP head already inside the checkpoint (text_config.mtp_num_hidden_layers: 1). No second model to download. Requires a checkpoint carrying mtp.fc.weight -- the -MTP build, not the plain -NVFP4 one (see above).

3. DFlash (separate draft model)

Add:

--speculative-config '{"method":"dflash","model":"<your-dflash-draft>","num_speculative_tokens":15}'

num_speculative_tokens: 15 is z-lab's own recommended value for this draft.

The published DFlash draft for this model family does not load on a released vLLM. z-lab/Qwen3.6-27B-DFlash (1,730,213,120 parameters across 58 tensors, 3,460,432,504 bytes) declares:

"layer_types": ["sliding_attention", "sliding_attention",
                "sliding_attention", "sliding_attention", "full_attention"],
"sliding_window": 2048

and _resolve_layer_attention in vllm/model_executor/models/qwen3_dflash.py raises on any mix of sliding and full:

NotImplementedError: DFlash does not yet support mixed sliding/full attention via
layer_types; see https://github.com/vllm-project/vllm/issues/40898.

The guard reads the draft's layer_types, not the target model's, so a stock unmodified Qwen3.6-27B target hits it identically. This is a vLLM gap, not a property of this checkpoint or of any particular target. z-lab's own card acknowledges it and directs users to install vLLM from the branch of that same PR:

uv pip install -U --torch-backend=auto \
  "vllm @ git+https://github.com/vllm-project/vllm.git@refs/pull/40898/head"

That is the upstream-sanctioned route, and if you can build from that branch it is the one to use.

Workaround used for the DFlash numbers below, for anyone pinned to a released vLLM: rewrite the draft's config.json so all five entries read "sliding_attention". The weight file is untouched: the draft used here has SHA-256 e0c050b34798d32728a164d2c3f1681746ff85c11945701b0205b654e2f1fdbe, which matches the upstream model.safetensors LFS hash exactly. Only config.json changed.

That mis-declares one layer: it was trained with full attention and is now given a 2048-token window. The target model verifies every drafted token by rejection sampling, so output correctness is unaffected either way; only acceptance can suffer. Note also that a sliding-window layer with a 2048-token window is identical to full attention below 2048 tokens of context, so for short contexts the declaration is exact rather than approximate. Every benchmark in this card ran below that threshold (see Limitations); acceptance beyond 2048 tokens of context is not measured here.

---

Speculative decoding is not a free speed-up

540 measurement cells: 3 serving configs x 3 workloads x 10 concurrency levels x 6 shuffled repeats, 9,126 streams. All 540 returned status ok, with 0 degraded, 0 failed, 0 errored streams. Run window 2026-07-27T20:49:22Z to 2026-07-28T09:39:38Z on one DGX Spark.

The three configs served the same NVFP4 weights, differing only by the single mtp.fc.weight tensor described above: the MTP rows come from the -MTP build, the base and DFlash rows from the plain -NVFP4 build. Reproducing the MTP rows requires a checkpoint that has the tensor.

Across the 90 (config, workload, concurrency) groups, the coefficient of variation of aggregate tok/s over the 6 repeats has median 0.42%, p90 2.41%, max 5.20%. Differences below a few percent in the tables below are not meaningful; the large ones are.

The three workloads

Chosen to differ in one thing: how predictable the next token is. Each pack generates 64 distinct prompts, so no two concurrent streams at N=64 share a prefix (a repeat would hit prefix caching and flatter the result).

Acceptance is the variable that explains everything

Acceptance rate is the fraction of drafted tokens the target model accepts, taken as a delta of vLLM's spec_decode_num_accepted_tokens_total / spec_decode_num_draft_tokens_total counters around each batch.

Mean acceptance across all ten concurrency levels:

(MTP on edit is 0.9998 to four decimal places; it rounds to 1.000 at every level.)

Acceptance is essentially flat in concurrency. It moves with content, not with load:

This is the point of the whole exercise, and it is the correction worth making to the usual framing: speculative decoding is not "a win at low concurrency that decays at high concurrency". It is a win where acceptance is high, and acceptance is a property of how predictable the output is. Concurrency governs how much spare compute is available to absorb rejected drafts, so it sets the size of the penalty, but the sign of the result is set by acceptance. Rewriting a file you were just given (edit) accepts almost everything; writing original fiction (prose) accepts less than half on MTP and 6% on DFlash. Same box, same flags, same day.

Aggregate throughput

Aggregate output tokens/second, mean of 6 repeats. max_tokens=512, temperature 0.8, thinking disabled; 8,996 of 9,126 streams ran the full budget and 130 stopped early. Token counts come from usage.completion_tokens, not SSE chunk counts (under speculation one chunk can carry several tokens, measured up to 13.1 tokens per chunk on DFlash, so chunk-counting under-reports badly).

prose

code

edit

What the tables say

• MTP wins at every concurrency level on code and on edit, including N=64: code +14.9% and edit +34.2% at N=64, with the largest margins at low concurrency (code +62.4%, edit +98.7% at N=1). Acceptance on those workloads is 0.763 and 1.000. There is no crossover to a loss.
• MTP is mixed on prose, the workload where its acceptance is lowest (0.475). It wins at N=1,2,4,6,8,24,32 and loses at N=12 (-11.4%), N=16 (-9.5%) and N=64 (-7.2%). At half-acceptance the drafting overhead and the win are close enough that the result flips with batch composition.
• DFlash is enormous on edit-shaped work: +716.1% single-stream (101.5 vs 12.4 tok/s) and still +39.4% at N=64, winning at every level, on 0.827 acceptance.
• DFlash hits a hard concurrency ceiling of about 21-22 streams on this box, and its flat lines past that are capacity, not speed. Reconstructing from the per-stream timings how many generations actually overlapped, base and MTP ran every level they were asked for (achieved/requested = 1.00 at all ten levels); DFlash reached 16 at N=16 and then stuck at 22, 22, 22 for N=24, 32 and 64. Everything above the ceiling queued. Confirmed live in the engine log: Running: 21 reqs, Waiting: 21 reqs, GPU KV cache usage: 97.2%. So DFlash prose reading 81.3 tok/s at N=64 against base's 353.5 is not a decode rate 77% slower - it is roughly 21 streams' worth of throughput while base runs 64. Judged per running stream the gap is far smaller; judged as a service it is worse than the number looks, because 43 of the 64 requests are waiting rather than generating. Not a scheduler setting: raising --max-num-batched-tokens from the spec-decode default of 2048 to 7296 moved the ceiling by one stream (22 to 21). It is KV capacity - see the next section.
• Because of that ceiling, DFlash's aggregate curve saturates on every workload, including the ones it wins: edit goes 389.8 tok/s at N=16 to 406.6 at N=64 (+4%) while base goes 146.1 to 291.8 (+100%). DFlash still wins edit at every level, but it wins by a shrinking margin for a structural reason rather than a workload one. On code it wins N=1 through N=12 and trails from N=16 up, which is exactly where the ceiling starts to bind.
• The governing variable is acceptance, not concurrency. The acceptance table above is flat across N; the outcome is not. Rank the three workloads by acceptance and you have ranked them by result, for both methods, at every level.

The KV cost nobody mentions

Speculative decoding takes memory out of the KV pool, which directly limits how much context you can serve. Read from the engine at load time (structural, not timed), at --max-model-len 262144 and --gpu-memory-utilization 0.72 on a 121.7 GiB unified-memory box:

MTP costs about 10% of the KV pool. DFlash costs about 40%, because it loads a second 3.46 GB model and needs its own KV. If your deployment is context-bound rather than throughput-bound, that is the number to budget against, and it is fixed regardless of workload.

This is also the cause of DFlash's ~21-stream ceiling, and the two costs compound. The pool is 40% smaller, and each running sequence additionally reserves lookahead slots for the 15 draft tokens it proposes per step, so the per-stream KV footprint roughly doubles: dividing the pool by the streams that actually fit gives about 31,900 tokens per stream under base and about 58,800 under DFlash. Fewer tokens shared among hungrier sequences is why the engine reports GPU KV cache usage: 97.2% at 21 concurrent requests and queues the rest. Budget for it directly: on this hardware, at these settings, DFlash serves roughly a third as many simultaneous streams as no-speculation does, whatever the workload. Both figures scale with --gpu-memory-utilization and shrink further at longer --max-model-len.

Per-stream fairness and TTFT at N=64

Aggregate throughput says what the box delivers; it says nothing about what one user waiting on one stream experiences. Per-stream tok/s at N=64 (min / p05 / p50 / p95 / max, mean of 6 repeats):

Base is almost perfectly uniform (max/min 1.02 on prose). Speculation makes streams unequal, because acceptance varies per stream: DFlash on code spans 5.08 to 12.28 tok/s in the same batch.

Time to first token is the clearest symptom of the ceiling, and it is worth reading as a separate cost from decode speed: waiting to start and generating slowly are different failures. Median TTFT at N=64: prose 1.9 s base / 2.6 s MTP / 138.1 s DFlash; code 2.3 / 3.1 / 74.9 s; edit 10.6 / 12.5 / 31.9 s. Base and MTP admit all 64 requests immediately, so their TTFT stays flat; DFlash's median request waits over two minutes for a slot, which no throughput figure shows.

Note this also explains why the DFlash per-stream numbers above look healthier than its aggregate: per-stream rate is measured over each stream's own decode window and therefore excludes queue time entirely. A DFlash stream is fast once it starts. Most of them just start late. If you are serving interactive traffic, size your concurrency to stay under the ceiling rather than relying on the queue.

---

Provenance and reproduction

Source. nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451: bf16, 1199 tensors, 55,457,998,304 bytes declared in its index. Per its own mergekit_config.yml it is a nuslerp merge of Qwen3.6-27B-Architect-Polaris2-Fable-B (weight 1.4) and Qwen3.6-27B-Architect-Polaris-Fable-F451 (weight 0.6), dtype: bfloat16. Uploaded with nightmedia's permission.

Architecture. Qwen3_5ForConditionalGeneration, model_type: qwen3_5. Hybrid attention: 64 decoder layers, 48 linear_attention (gated-delta / Mamba-style) and 16 full_attention, one full-attention layer every fourth (full_attention_interval: 4, at indices 3, 7, ... 63). Hidden size 5120, 24 attention heads, 4 KV heads, head dim 256, intermediate 17408. Vocabulary 248320, max_position_embeddings 262144. Vision tower present (language_model_only: false), so this is a vision-language checkpoint; MTP head is one decoder layer (mtp_num_hidden_layers: 1).

Quantization. NVIDIA TensorRT Model Optimizer 0.45.0 (producer: {"name": "modelopt", "version": "0.45.0"} in hf_quant_config.json), calling examples/llm_ptq/hf_ptq.py directly rather than the huggingface_example.sh wrapper, which never forwards a dataset flag and falls back to a gated default:

python hf_ptq.py \
  --pyt_ckpt_path='<bf16 checkpoint>' \
  --export_path='<out>' \
  --qformat=nvfp4 \
  --calib_size=512 \
  --batch_size=0 \
  --inference_tensor_parallel=1 \
  --dataset cnn_dailymail

Calibration: cnn_dailymail (public), 512 samples. Container: nvcr.io/nvidia/tensorrt-llm/release:spark-single-gpu-dev on the DGX Spark itself.

quant_algo: NVFP4, kv_cache_quant_algo: FP8, group_size: 16, weights and input activations both 4-bit float. ModelOpt selected 149 exclude patterns automatically: lm_head, model.language_model.embed_tokens, all 48 linear_attn.conv1d / in_proj_a / in_proj_b blocks, model.visual*, and mtp* / mtp.layers.0*. The linear-attention, vision and MTP paths stay at higher precision; the full-attention and MLP linears are the ones that went to NVFP4.

Sizes. Input 1199 tensors / 55,457,998,304 bytes bf16. ModelOpt export 2398 tensors / 20,487,991,904 bytes -- this is the plain -NVFP4 repo. The -MTP build is that same export plus mtp.fc.weight (104,857,600 bytes, carried through unquantized as BF16 exactly as vLLM expects), giving 2399 tensors / 20,592,849,504 bytes. The two differ by that one tensor and nothing else.

Benchmark harness. specbench.py (matrix runner), workloads.py (the three prompt packs), report.py / dashboard.py (charts), plus the raw 540-cell overnight.json with every per-stream record behind these tables:

TODO before publishing: replace with the harness repository URL. https://github.com/PLACEHOLDER/PLACEHOLDER

The harness is host-side, stdlib-only Python 3.12. It drives docker directly; records usage.completion_tokens rather than SSE chunks; flags any cell with a missing usage block or a generation that stopped short as degraded rather than reporting it as a clean number; and uses at least 64 distinct prompts per pack so no two concurrent streams share a prefix (a repeat would hit prefix caching and flatter the result). Reproducing the exact matrix:

python3 specbench.py --workloads prose,code,edit \
  --levels 1,2,4,6,8,12,16,24,32,64 --repeats 6 --out overnight.json

---

Credits

• nightmedia: the base merge, Qwen3.6-27B-Architect-Polaris2-Fable-B-F451, and its Architect-Polaris merge stages. All the model work is theirs. Uploaded with their permission. That merge in turn credits Qwen/Qwen3.6-27B, DavidAU's Qwen3.5/3.6-27B finetunes, and armand0e's Qwen3.6-27B-Fable-5-Experimental.
• Qwen (Alibaba): Qwen3.6-27B, the foundation everything above is built on. Apache-2.0.
• z-lab: Qwen3.6-27B-DFlash, the DFlash draft model measured here (MIT licence, per its own repo metadata). The weights used were byte-identical to theirs; only config.json was rewritten to work around a vLLM loader gap.
• NVIDIA: TensorRT Model Optimizer 0.45.0, which did the quantization, and the NVFP4 format.
• vLLM and the community Grace-Blackwell image ghcr.io/timothystewart6/vllm-gb10.

Licence: Apache-2.0, inherited from the base model, which declares license: apache-2.0, as does Qwen/Qwen3.6-27B. The DFlash draft is separately MIT-licensed by z-lab and is not redistributed here.

---

Limitations

Read these before generalising any number above.

• One box, one run. Every measurement comes from a single DGX Spark: GB10, sm_121a, arm64, 121.7 GiB unified memory shared between CPU and GPU. A discrete-GPU system with separate HBM will not reproduce the KV-pool arithmetic, and probably not the crossover points either. There was no second machine and no second run of the full matrix.
• One model. These are acceptance rates for this merge, whose MTP head was inherited through a mergekit merge and never retrained against the merged weights. Another Qwen3.6-27B derivative, or the stock Qwen release, will have a different MTP acceptance, possibly much better. Do not read 0.475 on prose as a property of MTP.
• Thinking was disabled. Every request passed chat_template_kwargs: {"enable_thinking": false}. Reasoning-mode output has different token statistics and would plausibly have different acceptance.
• Short sequences only. Prompts are 57-71 tokens (prose), 79-94 (code) and 820-849 (edit); generation was capped at 512 tokens. The longest sequence in the entire matrix is under 1,400 tokens. Nothing here measures behaviour at 32k, 128k or 262k context, and in particular the DFlash draft's 2048-token sliding window was never exceeded, so the layer_types rewrite described above was exact for these runs and is untested beyond 2048 tokens of context.
• Community image, unknown local patches. vLLM 0.25.2.dev0+g752a3a504.d20260721 inside ghcr.io/timothystewart6/vllm-gb10:v0.25.1-gb10.2. This is a third-party Grace-Blackwell build, not an official vLLM release; it may carry patches that affect these numbers. A stock vLLM on other hardware may differ.
• num_speculative_tokens was not swept. MTP ran at 2, DFlash at 15. Both are plausible defaults, neither was tuned per workload, and the crossover points would move if they were.
• Throughput only. The workload graders check that responses are complete and non-refusing; they do not score quality. This card contains no evidence about whether NVFP4 degrades output quality relative to the bf16 source, and no benchmark scores of any kind.
• Text-only testing. The vision tower is present and was left unquantized, and pipeline_tag is inherited from the base model, but every request in this card was text. Image and video input were not exercised at all.
• Format conversion, not a retrain. Whatever the original merge does well or badly, it still does. Its behaviour, style, biases and failure modes carry through unchanged.