Signal-3.8-27B-NVFP4

Signal-3.8-27B-NVFP4

NVFP4 (W4A16) quantization of agentionai/Signal-3.8-27B for NVIDIA Blackwell GPUs (RTX 50-series, DGX Spark, B100/B200), produced with NVIDIA TensorRT Model Optimizer 0.45.0.

Signal is a "directness" self-distill of Qwen/Qwen3.8-27B — only lm_head.weight differs from stock — that produces ~57% fewer answer tokens and ~52% fewer thinking tokens at matched answer quality. This quant keeps that behavior: same terse answers, ~4x smaller weights, NVFP4-accelerated decode on Blackwell.

Sampling — read this first

Use temperature 0.6, top_p 0.95, top_k 20, min_p 0.05. The upstream checkpoint originally shipped a generation_config.json with temperature: 1.0, which could degenerate on casual/open-ended prompts (garbled reasoning, noisy follow-ups). Upstream has since patched their default to 0.6 (their min_k: 0.05 appears to be a typo for the standard min_p, which is what this repo ships). The generation_config.json in this repo is already updated; make sure your server applies it (recent vLLM does by default; if you pass --generation-config vllm, set the sampling params yourself). Use sampling rather than greedy — upstream observed a generation loop at temperature 0.

Usage (vLLM)

vllm serve Shockem/Signal-3.8-27B-NVFP4 \
  --quantization modelopt --dtype bfloat16 \
  --reasoning-parser qwen3 \
  --max-model-len 65536 \
  --trust-remote-code

Chat template and tokenizer are the stock Qwen3.8 ones. Thinking is on by default; send "chat_template_kwargs": {"enable_thinking": false} to disable. Vision works as in the base model (add --language-model-only if you only need text).

Speculative decoding (MTP)

The checkpoint carries the Qwen3.8 MTP head (model_mtp.safetensors) plus an optional vocab-truncated draft head (40960-id BF16 mtp.draft_lm_head, mapping in mtp_draft_vocab_ids.pt) that raises draft acceptance ~20% at equal quality. Using the truncated draft head requires the syv-ai qwen3_5_mtp patch for vLLM; without the patch, simply serve without --speculative-config (or use stock MTP) — the truncated head is inert otherwise.

Quantization recipe

• Model Optimizer 0.45.0, NVFP4 W4A16: NVFP4 weights with block scales, BF16 activations
• ~400 linear tensors quantized; embeddings, vision tower, and MTP layers kept in BF16
• Dequantized-weight cosine similarity vs the BF16 source: 0.9955

Quality

Passed initial testing on a house agentic harness (instruction / reasoning / structured / coding / compaction suites, 10 runs each), served by vLLM v0.28 TP=2 on 2x RTX 5060 Ti 16 GB, 200k context, FP8 KV, MTP 3:

* 9 of 10 hard runs scored 94.8%; the single 83.2% run hit a known-flaky LRU codegen test (syntax), which passes on rerun. Decode ~50-70 tok/s with total solution wall time roughly half of stock Qwen3.8-27B thanks to Signal's terseness.

Optional: FP8 draft stack (+6.6% decode)

The MTP draft stack (mtp.* is excluded from the NVFP4 quantization, so it normally runs as BF16) can be converted to FP8 e4m3 for a measurable decode speedup with no output-quality change — the drafter only proposes tokens; the target model verifies every one. Measured on 2× RTX 5060 Ti (vLLM 0.28.0, TP=2, 200k ctx, FP8 KV, MTP 3, batch 1):

Converter script, patched qwen3_5_mtp.py (vLLM 0.28.0), and instructions: FP8 MTP draft stack gist. Note: the FP8 variant checkpoint requires the patched model file — stock vLLM will silently misload it, so keep the stock checkpoint as fallback.

Credits

• agentionai for the Signal fine-tune
• Qwen for the Qwen3.8-27B base model
• Quantized and published by Shockem

vLLM + MTP speculative decoding note

vLLM (as of v0.28) rejects min_p when speculative decoding (MTP) is enabled — requests will 400 with "min_p and logit_bias sampling parameters are not yet supported with speculative decoding". If you run this checkpoint with the MTP draft head on, either:

• drop min_p from your sampling params (the repetition_penalty: 1.05 default in this repo's generation_config.json already guards against the repetition runaways min-p is meant to prevent), or
• serve without speculative decoding to use min_p as upstream recommends.