Qwen3.8-Uncensored-NVFP4-MTP

Qwen3.8-27B-Uncensored NVFP4 (MTP)

Follow me on X @BennyDaBall_OG !

Want to support my work? Buy me a coffee on Ko-fi !

A native NVFP4 GGUF of JonathanColetti/Qwen3.8-27B-Uncensored, the Heretic-abliterated build of Qwen3.8-27B. The whole transformer backbone is quantized to 4-bit NVFP4 for Blackwell, and the MTP (multi-token prediction) speculative head is kept intact for fast decoding. No retraining, no distillation, just a clean quant.

---

What is this?

• 27B, uncensored, NVFP4. Every attention, Gated DeltaNet, and MLP weight across all 64 layers is quantized to native NVFP4 (GGML type 40). The lm_head, token embeddings, and the MTP draft head stay in BF16.
• MTP head retained. The model ships as 65 blocks (blk.64 is the MTP draft head), so llama.cpp can self-speculate with --spec-type draft-mtp for a large decode speedup, no separate draft model needed.
• Vision included. The repo ships the BF16 vision projector (mmproj-BF16.gguf), so the model sees images and video frames when loaded with --mmproj. The abliteration only modified text-model tensors, so the vision tower is stock Qwen3.8 quality.
• 262,144 native context.

Use this file when you want a fast, uncensored 27B on a Blackwell GPU (RTX 5090, 5080, RTX PRO) with native FP4 density and built-in speculative decoding.

---

The files

sha256 (model):  db17acbca53da5a7b0e861175b198cc6fd467865e99d1ad53d9dca584257a1a1
sha256 (mmproj): 5ac423f8a29059dc24e51bc6a43e9380dcd57a9347f28b62591e0b3f60b7081c

The corrected chat template is embedded in the model file. Text-only use works without the mmproj; add it when you want image input (~1 GiB extra VRAM).

---

Requirements

• A Blackwell (sm_120) GPU for the native FP4 path.
• A recent llama.cpp with NVFP4 CUDA kernels and the qwen35 architecture, or LM Studio 2.29.1+ (runtime llama.cpp-nvidia-cuda12 2.29.1 loads it and runs coherently).
• For the MTP speedup, a build with the draft-mtp speculative path.

---

Quick Start

llama.cpp / llama-server

llama-server \
  --model Qwen3.8-27B-Uncensored-NVFP4-MTP.gguf \
  --mmproj mmproj-BF16.gguf \
  --ctx-size 229376 \
  --flash-attn on \
  -ctk q8_0 -ctv q8_0 \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --spec-draft-p-split 0.2 \
  --temp 1.0 --top-p 0.95 --top-k 20

Measured on an RTX 5090 (131k context, q8 KV, greedy, 512-token generations, mean of 2 runs): 105.8 / 136.3 / 123.0 tok/s on prose / code / 12k-deep context versus 75.2 / 74.0 / 72.2 with speculation off, a 1.4-1.8x lift. MTP acceptance is hardware and prompt dependent. Sweep --spec-draft-n-max from 1 to 6 and keep whatever is fastest on your box. Drop the --mmproj line (or pass --no-mmproj) for a text-only server with a little more VRAM headroom.

LM Studio

Load the file and chat. Two things to know:

• Thinking is on by default. Give it a generous max-tokens budget or the whole budget can be spent inside the hidden reasoning block and the visible answer comes back empty. The old Qwen3 /no_think trick does not work on this template (LM Studio returns the whole budget as reasoning_content); on llama-server, disable thinking with chat_template_kwargs: {"enable_thinking": false}.
• MTP speculation works here too, but tune it. LM Studio (runtime 2.29.1+) detects the embedded MTP head and turns speculation on automatically, but at the default settings it drafts too short to gain any speed. Set max draft tokens 3 and continue probability 0.2, in the UI speculative decoding settings or at load time with lms load ... --speculative-draft-mtp --speculative-draft-max-tokens 3 --speculative-draft-min-continue-probability 0.2. Measured on an RTX 5090 (131k context, q8 KV, greedy, 512-token generations, prose / code / 12k-deep): speculation off 71.1 / 71.2 / 69.1 tok/s, tuned 79.9 / 88.9 / 89.3 tok/s. Short drafts win here; max tokens 2-3 with probability 0.1-0.2 is the sweet spot, and max tokens above 4 or probability 0.3+ is slower than speculation off. The llama-server draft-mtp path is faster still on the same file.

---

Performance (RTX 5090, single-stream)

Measured with the llama.cpp draft-mtp path, q8_0 KV cache, flash attention on. Single-run figures, not a formal benchmark.

MTP self-speculation reaches high draft acceptance on structured output (counting, code, repetitive passages) and lifts decode well above the no-speculation baseline. Acceptance and speedup drop on high-entropy creative prose, as expected.

---

What is NVFP4?

NVFP4 is NVIDIA's 4-bit floating-point weight format for Blackwell tensor cores: 16-element blocks, each with an FP8 (E4M3) block scale plus a global tensor scale. It keeps more of the weight distribution than integer 4-bit and runs on the FP4 tensor cores, so the whole backbone stays dense at 4-bit on a Blackwell card.

---

How it was made

1. Started from the JonathanColetti/Qwen3.8-27B-Uncensored BF16 checkpoint.
2. Converted to a BF16 GGUF parent (MTP head preserved as blk.64).
3. Quantized the transformer backbone to NVFP4 with an importance matrix, keeping lm_head, embeddings, and the MTP head in BF16.
4. The vision projector is the base model's own BF16 projector (the abliteration never touched the vision tower), verified working against this quant (image in, correct description out).

No weights were trained or fine-tuned. The abliteration and all behavior come from the base model; quantization is a transformation only.

---

Notes

• Uncensored / abliterated. Refusal behavior is inherited from the base model (Heretic abliteration), not from this quant. You are responsible for how you use it.
• Blackwell only for the native FP4 path.

---

Acknowledgements

• Alibaba / Qwen for Qwen3.8-27B.
• JonathanColetti for the uncensored base, Qwen3.8-27B-Uncensored.
• The maintainers behind llama.cpp and LM Studio.

Apache-2.0, same as every upstream artifact. "Qwen" is a trademark of Alibaba, used only to identify the upstream model; this repo is not affiliated with or endorsed by Alibaba.

Quantized locally by BennyDaBall.

Follow me on X @BennyDaBall_OG !

Want to support my work? Buy me a coffee on Ko-fi !