Tess-4-27B-FP8

Tess-4-27B-FP8

FP8 (compressed-tensors, FP8_DYNAMIC W8A8) quantisation of migtissera/Tess-4-27B, attnbf16 variant: the entire self-attention path is kept in bf16 and only the MLPs are FP8. Ships a working MTP self-speculative-decoding head.

What's kept in bf16

lm_head, the MTP head, the vision tower, the whole linear_attn (Gated-DeltaNet / SSM) block, and the entire self_attn path (incl. the attention output gate fused into q_proj on Qwen3.5/3.6). Only the ~17 B MLP params are FP8. This keeps quantisation off the multiplicative attention gate and the 16 long-range full-attention layers, at a cost of ~+1.5 GiB (~4.6 %) vs a fully-quantised FP8 build. See recipe.yaml.

Accuracy vs the bf16 parent: KLD ≈ 0.0078 nats (kld_tess_attnbf16.json), measured per-token on neuralmagic/calibration (8 samples, seq 1024). This is markedly lower than a plain-attention FP8 build of the same base.

Speculative decoding (MTP)

The bf16 MTP head is declared in quantization_config.ignore so vLLM loads it correctly. (A bf16 MTP head regrafted into a compressed-tensors quant is otherwise mis-loaded and yields 0 % draft acceptance — this build fixes that.)

Measured MTP acceptance: 75.2 % (vLLM 0.26.0, Blackwell, greedy, --speculative-config '{"method":"mtp","num_speculative_tokens":3}').

vllm serve huginnfork/Tess-4-27B-FP8 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --max-num-seqs 32

--max-num-seqs 32 (or lower) is required — Qwen3.6 is a hybrid linear-attention model whose Mamba cache otherwise runs out of blocks at the default max_num_seqs.