Qwen3.8-27B-NVFP4-RTX5090-LMHead4

⚠️ Superseded

The main checkpoint now has the NVFP4 lm_head and the MTP head removed — it is strictly better than this variant (17.92 GB, 2 shards).

Use Qwen3.8-27B-NVFP4-RTX5090 instead. This repo stays up so existing pins keep working.

---

Qwen3.8-27B-NVFP4-RTX5090-LMHead4

Qwen3.8-27B-NVFP4-RTX5090 with lm_head quantized to NVFP4 instead of left in BF16. Every other tensor is bit-identical to the parent.

Smaller and measurably faster, with no accuracy or acceptance cost.

2.23× the decode of the unquantized-lm_head parent without speculation, on one RTX 5090.

---

Why this is free speed

At concurrency 1 this model is weight-bandwidth bound, not compute bound. Measured on the parent: 18.80 GiB of resident weights at 81.6 tok/s implies a 1.65 TB/s read rate against the RTX 5090's 1.79 TB/s spec — about 92% of peak. Decode is streaming the whole model once per token.

lm_head is a full-vocabulary (248,320 × 5,120) GEMM evaluated on every token, so it is 2.54 GB of that per-token read. Quantizing it to NVFP4 cuts it to 0.72 GB.

The bandwidth model predicted 89.7 tok/s; measured 88.45 — within 1.4%. On this hardware, bytes removed from the weight read path convert almost linearly into tokens per second.

Note the distinction: embeddings are the same 2.54 GB but are a gather (~10 KB/token), so quantizing them would save capacity, not decode speed. lm_head is the only large BF16 block on the per-token critical path.

---

Accuracy

Same 20 items per task, seed 20260815, thinking on, temperature=1.0, top_p=0.95, 24k generation cap — both arms run through the same harness on SGLang.

Read this as "no degradation", not "an improvement." At n=20 per task with temperature=1.0, a 4-item difference is well inside noise, and 6–9 items per task hit the 24k truncation cap on both arms. The claim supported by this data is that 4-bit logits did not measurably hurt quality — not that they helped.

These numbers are also not comparable to the parent card's published smoke, which was measured on vLLM 0.27.1; this harness runs on SGLang. Both arms here share one harness, so the comparison between them is valid.

Speculative decoding

The v2 DSpark drafter was trained against the parent's logits, so changing lm_head could have degraded acceptance. It did not — acceptance is flat to slightly up:

Acceptance measured per request with the prefix cache flushed, from cumulative verify counters.

---

Serve

Identical to the parent — no flag changes:

sglang serve --model-path gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090-LMHead4 \
  --trust-remote-code --tp-size 1 \
  --context-length 65536 --kv-cache-dtype fp8_e4m3 \
  --attention-backend flashinfer --chunked-prefill-size 2048 \
  --mamba-radix-cache-strategy extra_buffer_lazy --mamba-ssm-dtype bfloat16 \
  --mem-fraction-static 0.90 --max-running-requests 2 \
  --speculative-algorithm DSPARK \
  --speculative-draft-model-path gittensor-model-hub/Qwen3.8-27B-DSpark-NVFP4 \
  --speculative-dspark-block-size 7 --speculative-draft-model-quantization modelopt_fp4 \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder

The native MTP head is retained, so vLLM speculation still works.

---

What changed

lm_head.weight (BF16, 248320 × 5120) replaced by four NVFP4 tensors:

lm_head.weight          U8        (248320, 2560)   packed E2M1
lm_head.weight_scale    F8_E4M3   (248320,  320)   per-group-16 scales
lm_head.weight_scale_2  F32       ()               global weight scale
lm_head.input_scale     F32       ()               activation scale, amax 49.75

The activation scale comes from measuring real lm_head inputs on the served parent (amax 49.75, p50 40.0 — a clean distribution, max/p50 = 1.24, no outlier tail). ModelOpt 0.45 performed the quantization so the E2M1 packing and UE4M3 scale layout match what the runtime expects.

lm_head was also removed from both exclude lists — hf_quant_config.json → quantization.exclude_modules and config.json → quantization_config.ignore. Missing the second one causes a silent fallback to an unquantized layer, surfacing as Parameter lm_head.input_scale not found in params_dict and then a 5120-vs-2560 shape error.

License

Apache 2.0, same as the parent and the Qwen3.8-27B base model.