Qwen3.8-27B-NVFP4-RTX5090-No-MTP

⚠️ Superseded

The main checkpoint now ships with the MTP head removed and an NVFP4 lm_head — it is strictly better than this variant (17.92 GB, 2 shards).

Use Qwen3.8-27B-NVFP4-RTX5090 instead. This repo stays up so existing pins keep working.

---

Qwen3.8-27B-NVFP4-RTX5090-No-MTP

Qwen3.8-27B-NVFP4-RTX5090 with the native MTP draft head removed. The target model is bit-identical — same NVFP4 weights, same calibration, same chat template. Only the 15 mtp.* tensors are gone.

---

Read this before choosing it

It saves download size, not memory. Measured on an RTX 5090, SGLang reports mem usage=18.80 GB for weights with and without the head — byte-for-byte the same, because mtp.* is namespaced outside the target graph and is never allocated unless you turn speculation on. If you are hoping to free VRAM, this variant will not do that.

It removes speculative decoding on vLLM entirely. DSpark is a SGLang-only algorithm, so on vLLM the native MTP head is the only draft path. Without it you decode at ~81.6 tok/s instead of ~136.9.

So this variant makes sense if you are SGLang-only and using the DSpark drafter (which is both faster and smaller than MTP anyway), or if you are shipping the weights somewhere that 0.85 GB matters. Otherwise use the parent — the head is free when idle.

Which drafter should I use anyway?

Even on the parent checkpoint, the MTP head is not the best option:

DSpark wins on speed, acceptance and size — the MTP head's value is that it works on vLLM, where DSpark does not exist.

---

Serve

SGLang with the DSpark drafter (recommended for this variant):

sglang serve --model-path gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090-No-MTP \
  --trust-remote-code --tp-size 1 \
  --context-length 65536 --kv-cache-dtype fp8_e4m3 \
  --attention-backend flashinfer --chunked-prefill-size 2048 \
  --mamba-radix-cache-strategy extra_buffer_lazy --mamba-ssm-dtype bfloat16 \
  --mem-fraction-static 0.90 --max-running-requests 2 \
  --speculative-algorithm DSPARK \
  --speculative-draft-model-path gittensor-model-hub/Qwen3.8-27B-DSpark-NVFP4 \
  --speculative-dspark-block-size 7 --speculative-draft-model-quantization modelopt_fp4 \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder

vLLM for the full 262,144-token window (no speculation available on this variant):

vllm serve gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090-No-MTP \
  --quantization modelopt --kv-cache-dtype fp8 --trust-remote-code \
  --max-model-len 262144 --max-num-seqs 16 --gpu-memory-utilization 0.97 \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml

Do not pass --speculative-config '{"method":"mtp"}' to this variant — the head is not present.

---

What changed, exactly

Removed 15 tensors, all under the mtp. prefix:

mtp.fc.weight                     mtp.layers.0.self_attn.{q,k,v,o}_proj.weight
mtp.norm.weight                   mtp.layers.0.self_attn.{q,k}_norm.weight
mtp.pre_fc_norm_embedding.weight  mtp.layers.0.mlp.{gate,up,down}_proj.weight
mtp.pre_fc_norm_hidden.weight     mtp.layers.0.{input,post_attention}_layernorm.weight

They were BF16, never quantized (the parent's hf_quant_config.json already excluded mtp*). Alongside the tensor removal:

• config.json — text_config.mtp_num_hidden_layers 1 → 0; mtp_use_dedicated_embeddings dropped
• hf_quant_config.json — exclude_modules 149 → 147 (mtp*, mtp.layers.0* removed)
• re-sharded 3 → 2 (the third shard held only mtp.* and became empty), index and crc32.txt regenerated

Everything else — tokenizer, chat template, preprocessor configs, generation config — is copied unchanged from the parent.

Verified: loads in SGLang as Qwen3_5ForConditionalGeneration, quant_algo=NVFP4, and answers correctly.

---

Accuracy, context and recipe

Unchanged from the parent, since the target weights are identical. See the parent model card for the full quantization recipe, 262K context behaviour, accuracy smoke and chat-template documentation.

License

Apache 2.0, same as the parent and the Qwen3.8-27B base model.