Qwen3.8-27B-Dominatrix-MTP-NVFP4

Qwen3.8-27B-Dominatrix-NVFP4

A mixed-precision NVFP4/FP8 quantization of allura-org/Qwen3.8-27B-Dominatrix, with the base model's BF16 multi-token-prediction (MTP) head grafted back in for speculative decoding.

Not-for-all-audiences. This is a quant of a roleplay/creative-writing finetune with mature content; inherits all behavior and content characteristics of the source model.

Attribution

• Finetune: allura-org/Qwen3.8-27B-Dominatrix (Apache 2.0), trained by Allura with Axolotl/Unsloth. All credit for the model's behavior, style, and training data goes to the original authors.
• Base model: Qwen/Qwen3.8-27B (qwen3_5, hybrid gated-linear-attention + full-attention, 64 layers, vision-capable).
• Quantization recipe format: mirrors RadixArk/Qwen3.8-27B-NVFP4's mixed-precision ModelOpt export (same producer version, same per-layer-type precision assignment), applied here to the Dominatrix finetune instead of the base model.

This repo only contributes the MTP graft and the quantization; no retraining was performed.

What was done

1. MTP graft. The Dominatrix finetune ships without a multi-token-prediction head (dropped during finetuning). The 15 BF16 MTP tensors (mtp.*) were copied over unmodified from the pre-finetune base checkpoint and merged into the finetune's safetensors shards, since no finetuned MTP head exists. This gives up speculative-decode acceptance rate slightly relative to a co-trained head, but MTP is draft-only — it does not change the served model's output distribution.
2. Quantization. NVIDIA ModelOpt 0.47.0.dev0, PTQ with method: max calibration on 1024 samples of abisee/cnn_dailymail (seq len 512).

Result: 2194 total tensors — 193 NVFP4 + 208 FP8 quantized layers, 15 MTP tensors carried at full BF16 precision. Structurally verified against the RadixArk reference export: identical per-layer precision map, byte-identical config.json quantization metadata, 0 missing / 0 extra tensors.

Quality: KL divergence vs. the BF16 finetune

Measured directly rather than assumed. Full-vocabulary next-token KL divergence between this NVFP4/FP8 checkpoint and the original BF16 Dominatrix weights, evaluated on 256 held-out cnn_dailymail samples (512 tokens each, ~120k scored token positions), using the exact same calibration recipe/data/seed reconstructed in-process (deterministic max calibration, so the reconstructed quantizer scales match this export's stored scales exactly wherever directly comparable, e.g. the FP8 self_attn projections).

This is a fairly aggressive quantization scheme relative to weight-only alternatives — activations and lm_head are quantized, not just weights, plus an FP8 KV cache — so a ~5% PPL increase is in the expected range for the precision budget, not a red flag.

Serving

Verified serving on SGLang with real FP4/FP8 kernels, FP8 KV cache, and NEXTN speculative decoding against the grafted MTP head:

--speculative-algo NEXTN --speculative-num-steps 3 \
--speculative-eagle-topk 1 --speculative-num-draft-tokens 4