Tess-4-27B-NVFP4A16

Tess-4-27B-NVFP4A16

NVFP4A16 W4A16 quantisation of migtissera/Tess-4-27B — a dense 27B qwen3_5 reasoning fine-tune (multimodal vision tower + hybrid linear/full attention) with an MTP speculative-decoding head. Produced with llm-compressor.

Quantisation

• Scheme: NVFP4A16 W4A16 — 4-bit FP4 weights (FP8 E4M3 per-group scales, group_size=16) with bf16 activations.
• Tooling: llm-compressor 0.12.0 + compressed-tensors 0.17.1, transformers 5.8.1, on a Blackwell RTX PRO 6000 (SM120).
• Quantised: the self_attn and mlp (gate/up/down) projections of the 64 language-model layers.
• Kept in bf16 (quantization_config.ignore): lm_head, all model.visual.* (vision tower), the entire linear_attn Mamba/SSM block (in_proj_*, out_proj, conv1d), and mtp.*. Keeping the SSM block in bf16 is essential on this family — NVFP4-ing it inflates KLD by ~2×.
• Recipe: recipe.yaml.

Speculative decoding (MTP)

Tess-4-27B ships the 15 mtp.* tensors, and they are byte-identical to the Qwen/Qwen3.6-27B base MTP head — the fine-tune reused it rather than retraining. llm-compressor drops MTP during quantisation (transformers v5 doesn't instantiate the submodule), so it was regrafted in bf16 from the source checkpoint and the 15 tensors are present in this repo's safetensors index for vLLM/SGLang to pick up.

Measured with vLLM (method: qwen3_5_mtp, num_speculative_tokens: 1, greedy, 12 mixed chat/code/reasoning prompts): The fine-tune shifted the trunk so far that the inherited head is useless for speculation: 0.0% of drafted tokens were accepted, and speculative decoding is counterproductive (it slows generation, since every rejected draft is wasted compute). Serve without --speculative-config. The mtp.* tensors are still shipped (matching the parent's structure and available for anyone who retrains the head), but do not enable MTP speculative decoding as-is. Positive control: the base Qwen/Qwen3.6-27B MTP head accepts 92.7% on the identical vLLM stack, so the number below is a genuine property of Tess's fine-tuned trunk, not a serving artefact. End-to-end throughput was 43.6 tok/s with speculation on vs 52.3 tok/s off = 0.83×.

Quality

KLD is per-token KL divergence vs the bf16 Tess-4-27B parent over 8 neuralmagic/calibration samples (max_seq=1024) — it isolates quantisation loss (not the fine-tune delta). PPL is wikitext-2-raw (test), non-overlapping 2048-token chunks, identical tokenizer to the parent. Compressed weights were decompressed to bf16 for the eval (the quant→dequant round-trip is applied, so precision loss is captured).

NVFP4A16 keeps activations in bf16, so the measured KLD below is exact for this scheme. NVFP4 on Qwen is always W4A16: the Qwen3.5 family is very sensitive to activation quantisation, and W4A4 roughly doubles KLD for no on-disk size benefit.

Inference

This is a dense checkpoint, so it runs under both 🤗 transformers and vLLM.

transformers

from transformers import AutoModelForImageTextToText, AutoProcessor
import torch

model = AutoModelForImageTextToText.from_pretrained(
    "lflare/Tess-4-27B-NVFP4A16", dtype=torch.bfloat16, device_map="auto",
)
proc = AutoProcessor.from_pretrained("lflare/Tess-4-27B-NVFP4A16")

vLLM (with MTP speculative decoding)

vllm serve lflare/Tess-4-27B-NVFP4A16 \
    --trust-remote-code --quantization compressed-tensors --max-model-len 8192 \
    --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":1}'

On Blackwell (RTX PRO 6000 / B200, SM120/SM100): FlashInfer's JIT arch probe currently mis-detects SM120 and aborts engine startup. Disable it:

TORCH_CUDA_ARCH_LIST=12.0+PTX VLLM_USE_FLASHINFER_SAMPLER=0 VLLM_USE_FLASHINFER=0 \
    vllm serve lflare/Tess-4-27B-NVFP4A16 --trust-remote-code --quantization compressed-tensors --max-model-len 8192