Huihui-Qwen3.8-27B-abliterated-NVFP8

Huihui-Qwen3.8-27B-abliterated — NVFP8

Block-16, two-level-scaled FP8 E4M3. 55.6 GB → 32.8 GB (−41%).

Stock vLLM cannot load this

NVFP8 is not an NVIDIA-defined format. NVIDIA's microscaling family is NVFP4 (block-16, E4M3 scales) at 4 bits and MXFP8 (block-32, E8M0 scales) at 8 bits — there is no vendor NVFP8. vLLM dispatches kernels by matching quantization args: _is_nvfp4_format requires num_bits == 4, _is_mxfp8 requires group_size == 32. This checkpoint's signature — 8-bit, tensor_group, group_size=16 — matches nothing, and loading raises NotImplementedError.

The nvfp8/ package in this repo supplies the Triton kernel and the dispatch hook that make it loadable.

Usage

The runtime ships in this repo, so download it as a directory rather than letting vLLM stream the weights into the HF cache — you need a local path to install the plugin from.

hf download trailio/Huihui-Qwen3.8-27B-abliterated-NVFP8 --local-dir ./qwen-nvfp8
cd qwen-nvfp8
pip install -e .        # registers a vllm.general_plugins entry point

from vllm import LLM
llm = LLM("./qwen-nvfp8", trust_remote_code=True, max_model_len=8192)
print(llm.generate(["The capital of France is"])[0].outputs[0].text)

pip install -e . has to run from that downloaded directory — it is what registers the entry point. Passing the repo id straight to LLM(...) without installing gets you the NotImplementedError described above, because the weights land in the HF cache and the plugin is never installed.

The entry point is required rather than a convenience: vLLM forces the spawn start method once CUDA is initialized, so calling register() by hand in the parent process never reaches EngineCore. Installation is what makes registration happen in every engine and worker process.

Needs ~31 GB of disk for the download and ~35 GB of VRAM at runtime (weights plus activations), before KV cache.

Requires sm_89+ (Ada or newer). Validated on RTX PRO 6000 Blackwell (sm_120, 96 GB) with vLLM 0.27.1.

Format

weight               [N, K]      float8_e4m3fn
weight_scale         [N, K/16]   float8_e4m3fn   one per 16 input elements
weight_global_scale  per-shard   float32

Weight-only (W8A16); activations stay bf16.

The global scale is per-shard rather than per-tensor because vLLM fuses gate_proj+up_proj into gate_up_proj and q/k/v_proj into qkv_proj. Those tensors were quantized independently and carry different global scales, so the value is expanded per-output-row at load time. Collapsing it to a single scalar mis-scales one shard by the ratio between them.

Group scales are rounded up to the next representable e4m3 value, not to nearest. Round-to-nearest lets a scale land below its target, pushing the group's largest element past 448 so it clips — measured at 3.4% of all elements. Rounding up drives that to zero.

What was quantized

400 Linear layers, 24.33B of 27.78B params (87.6%). Left in BF16:

All quantized K-dims (5120, 6144, 17408) divide by 16, so groups tile with no padding.

Accuracy

Weight round-trip error, identical tensors:

On outlier-heavy weights: 0.0213 vs 0.0249 per-channel. 0.025 is the E4M3 floor — 3 mantissa bits — not a property of the scaling scheme.

Triton kernel error against an fp32 reference is ≤3.7e-3 across every layer shape at M=1 through M=2048, roughly an order of magnitude below the quantization error itself.

PTQ with no calibration data: weight-only NVFP8 derives its scales from the weights, so there is no dataset dependence and nothing to overfit.

Known issues

On containers with CUDA < 12.9, FlashInfer's JIT cannot resolve sm_120 and fails with FlashInfer requires GPUs with sm75 or higher. Set VLLM_USE_FLASHINFER_SAMPLER=0 or use a CUDA ≥ 12.9 image. Unrelated to quantization — it affects the sampler, not the model.

Provenance

Quantized from huihui-ai/Huihui-Qwen3.8-27B-abliterated, itself an abliterated derivative of Qwen/Qwen3.8-27B.

Abliteration removes refusal behavior; the base model's reduced-filtering caveats apply here unchanged. Quantization preserves behavior — it neither adds nor removes safety properties.