Qwen3.8-27B-heretic-ara-NVFP4

Qwen3.8-27B-heretic-ara-NVFP4

An NVFP4 (4-bit floating point) quantization of trohrbaugh/Qwen3.8-27B-heretic-ara (itself a Heretic ARA decensor of Qwen/Qwen3.8-27B).

Produced with NVIDIA TensorRT Model Optimizer (Spark Foundry PTQ, vlm_ptq / qformat=nvfp4, 512 calibration samples, sequence length 2048) so the model fits and runs on a single 128 GB DGX Spark (GB10) through vLLM's Blackwell FP4 path.

No training, fine-tuning, or weight edit was done here except the two steps below. The heretic / ARA work is entirely upstream. This repo is a numeric format conversion of that finished checkpoint, plus an MTP-head repair after ModelOpt dropped the draft tensors.

It is a lossy conversion. No quality eval of the 4-bit result against the bf16 source was run. Treat upstream capability claims (Qwen's, and trohrbaugh's refusal-rate numbers) as claims about the bf16 weights, not as verified properties of this quantization.

---

What we did

1. Download trohrbaugh/Qwen3.8-27B-heretic-ara (51.8 GB bf16).
2. PTQ to NVFP4 with NVIDIA ModelOpt on a DGX Spark, Foundry path, vlm=true (vision tower left in higher precision), 512 calib samples, seq_len 2048.
3. Restore the MTP head. ModelOpt exported text_config.mtp_num_hidden_layers: 1 but wrote zero of the 15 mtp.* tensors, including mtp.fc.weight. That is the known silent-failure shape: vLLM will load, draft from an uninitialised projection, and only get slower. We spliced all 15 mtp.* tensors unchanged (bf16) from the source checkpoint into an extra shard (model-mtp-fc.safetensors, 849.4 MB) and listed those modules in quantization_config.ignore so they are not read as NVFP4.
4. Nothing else. Same tokenizer, chat template, vision processor files (borrowed from a same-arch donor at register time where ModelOpt omitted them). Language-model and vision-tower tensor names match the official PassingByPixels/Qwen3.8-27B-NVFP4 export except the restored MTP set.

---

What this quantization is

NVFP4 stores each weight as an E2M1 4-bit float (1 sign, 2 exponent, 1 mantissa bit), with an 8-bit scale shared across each block of 16 weights.

Mixed precision, by design. ModelOpt leaves these in higher precision:

---

MTP validation (this artifact)

Served on DGX Spark GB10 with vLLM 0.25.1-gb10.2:

--speculative-config '{"method":"mtp","num_speculative_tokens":2}'

Smoke generation (96 tokens, temperature 0): spec_drafted=86, spec_accepted=53, accept rate 0.616. If the head were missing or random, drafted would be 0 or accept would sit near 0. That is the functional check. It is not a quality bench.

---

Source

• Quantized from: trohrbaugh/Qwen3.8-27B-heretic-ara
• That model is a Heretic (custom fork) v1.2.0+custom ARA decensor of Qwen/Qwen3.8-27B
• Official (non-heretic) NVFP4 of the same architecture: PassingByPixels/Qwen3.8-27B-NVFP4

---

Serve (vLLM)

vllm serve PassingByPixels/Qwen3.8-27B-heretic-ara-NVFP4 \
  --served-model-name qwen3.8-27b-heretic-ara-nvfp4 \
  --kv-cache-dtype fp8 \
  --attention-backend flashinfer \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.72 \
  --reasoning-parser qwen3 \
  --trust-remote-code \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

Thinking mode is on by default. Disable per request with chat_template_kwargs.enable_thinking: false.

---

Limitations

• Lossy 4-bit conversion. No side-by-side quality eval vs the bf16 heretic source.
• No throughput sweep was run on this repo (the table on the official Qwen3.8-27B NVFP4 card is a different checkpoint).
• Client max_tokens must be less than the served max_model_len.
• MTP + FlashInfer on GB10 has a known crash class at some GQA / k settings; this checkpoint was loaded and drafted at k=2 with FlashInfer on v0.25.1-gb10.2.

---

License

Apache 2.0, inherited from trohrbaugh/Qwen3.8-27B-heretic-ara and Qwen/Qwen3.8-27B.