Qwen3.8-27B-huihui-abliterated-NVFP4-NInfer

Qwen3.8-27B abliterated — NVFP4 for NInfer

An NVFP4 artifact of huihui-ai/Huihui-Qwen3.8-27B-abliterated, packaged for the NInfer engine.

This is the refusal-removed (abliterated) counterpart to the official neroued/Qwen3.8-27B-NInfer artifact. It carries the complete container — Text, Vision, MTP, the optimized proposal head, tokenizer, chat template, generation config and media-processor resources — as one file.

qwen3_8_27b_huihui_abliterated_nvfp4.ninfer
  21,492,695,040 bytes
  sha256 02c0c80616e2dd353133355d840aa6418d83f4c523369ad93b426e6c5bbc83c8
  identity  qwen3.8-27b / nvfp4      container v2      1124 objects

Why this exists

Qwen3.8-27B is the king of the local models right now, and the great work Neroued has done with the NInfer engine runs it extremely quickly on Blackwell. There has also been a lot of work on obliterating and otherwise liberating models, including Pliny's OBLITERATUS suite.

Some abliterated models have been "ninferized" already — see Related work below. Two things I hadn't seen:

• the walkthrough/recipe to do it yourself
• Pliny's OBLITERATUS Qwen3.8-27B, ninferized

Both are offered here. The recipe is in AGENT.md, SUPPORTED.md and scripts/ — a triage script that tells you in seconds whether a checkpoint is convertible, ten documented failure modes with the measurement behind each, and nine scripts proven by running the pipeline end to end on a second checkpoint. Under $5 of RunPod time converts a compatible checkpoint; measured runs came in at $1.69 and $3.30.

What it is, precisely

Format conversion only. No fine-tuning, no re-abliteration, no behavioural editing was performed. All refusal characteristics are inherited unchanged from the upstream huihui-ai checkpoint, which removes the refusal direction by weight editing (per remove-refusals-with-transformers, the Arditi et al. 2024 family) on layers 18–51 only, leaving layers 0–17 and 52–63, the vision tower, and the MTP head untouched.

Structurally identical to the official artifact

Diffed object-by-object against the official qwen3.8-27b/nvfp4 artifact:

identity equal        : True     qwen3.8-27b/nvfp4
file_bytes equal      : True     21,492,695,040
payload_offset equal  : True     180,224
object count equal    : True     1124  (1118 tensors + 6 resources)
names + order equal   : True
objects differing in (name, format, layout, length, shape) : 0

Every object sits at the same offset with the same numeric format, layout, byte length and shape. Only the values differ. KV-cache behaviour, memory profile and compute path are identical, so it is a drop-in replacement for the official NVFP4 artifact with no flag changes.

Numeric allocation

in_proj_a / in_proj_b stay BF16 because of the CUTLASS N % 64 == 0 constraint.

Requirements

• Blackwell GPU, compute capability 12.0 (sm_120a). NVFP4 will not run on Ada or Hopper. NInfer's build hard-rejects any other architecture (CMakeLists.txt).
• ~19.0 GiB VRAM for weights. A 32 GB card leaves ~13 GiB for KV cache.
• Stock NInfer — no engine patch required. The qwen3.8-27b/nvfp4 identity is registered upstream (this artifact loads on an unmodified build). Earlier NVFP4 artifacts needed a patch because that identity predated issue #25; that is no longer the case.

Quick start

ninfer qwen3_8_27b_huihui_abliterated_nvfp4.ninfer \
  --prompt "Explain NVFP4 in three sentences." \
  --max-context 16384 --max-new 256 \
  --spec mtp --draft-tokens 5 --lm-head-draft

Serving:

ninfer-serve qwen3_8_27b_huihui_abliterated_nvfp4.ninfer \
  --model-id qwen3.8-27b-huihui-abliterated \
  --max-context 262144 --kv-capacity auto --max-concurrency 1 \
  --kv-dtype nvfp4 --spec mtp --draft-tokens 5 --lm-head-draft \
  --preserve-thinking --vision \
  --host 0.0.0.0 --port 8080

--model-id is optional but recommended: it labels this model distinctly in API responses. The artifact's on-disk identity is qwen3.8-27b/nvfp4, the same as the official artifact, so without it the two are indistinguishable to any client.

Measured

Both profiles of this abliterated checkpoint, same GPU, identical flags, --max-context 16384, --kv-capacity auto, greedy.

NVFP4 is ~35 % faster on speculative decode and 1.6–2× on prefill, for 3.1 GiB more VRAM. Non-MTP decode is a wash.

Scope of validation — read this

What was verified: the artifact loads; the identity is accepted by stock NInfer; MTP speculative decoding engages with high acceptance and zero fallbacks; thinking mode works; output is coherent and correct on reasoning, code and explanation prompts; the container is structurally byte-for-byte equivalent to the official artifact; the source checkpoint is intact (1199 tensors, 0 non-finite values).

What was not run: no capability benchmark, no fidelity comparison against BF16, and no refusal-rate measurement. No HumanEval+/GPQA numbers are claimed here because none were produced. Abliteration has a documented capability cost, and this artifact inherits whatever that cost is for the upstream checkpoint — unquantified. If you need those numbers, measure them.

Verify what you downloaded

sha256sum -c SHA256SUMS
python3 -m tools.artifact.inspect qwen3_8_27b_huihui_abliterated_nvfp4.ninfer

Expect identity qwen3.8-27b/nvfp4, objects 1124 (1118 tensors, 6 resources), file_bytes 21492695040.

Files

Provenance

base       huihui-ai/Huihui-Qwen3.8-27B-abliterated @ 739e3c5b89849f6c238ce1e5b70008612ae42cdd
             itself an abliteration of Qwen/Qwen3.8-27B @ 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
quantized  local: llmcompressor 0.13.0 / compressed-tensors 0.18.0
             recipe taken verbatim from unsloth/Qwen3.8-27B-NVFP4 config.json
             calibration: ultrachat-200k, 32 samples, seq 2048
converter  github.com/Neroued/ninfer @ a140e7ae82a11ed2f370a4d8f2cc16268a3790b8
             + patches/0001-nvfp4-provenance.patch
runtime    minimum_revision 52320554b5e71a9da96bff809ddf67ac5773ed63, cuda_architecture sm_120a

The included patch changes four provenance constants only — BASE_REPOSITORY, BASE_REVISION, QUANTIZED_REPOSITORY, QUANTIZED_REVISION in tools/convert/qwen3_8_27b/recipe_nvfp4.py. Those values are read in exactly one place, where they are written into the conversion report; they gate nothing. Without the patch the report would claim Qwen/Qwen3.8-27B and unsloth/Qwen3.8-27B-NVFP4 provenance this artifact does not have. It is an honesty fix, not a validation bypass. No check anywhere in the conversion path was disabled or relaxed.

Also available

The same source checkpoint converted to the groupwise-int profile is published at Barding-Defense/Qwen3.8-27B-huihui-abliterated-groupwise-int-NInfer.

qwen3_8_27b_huihui_abliterated.ninfer
  18,210,531,328 bytes
  sha256 8c9f9d67a07ac97506978f6db6695d8074f78dec0fb80c4a85a8fb6fbedd7f03
  identity  qwen3.8-27b / groupwise-int

Q4/Q5/Q6 text body with W8G32_F16S embedding and output head. Smaller (15.9 GiB resident vs 19.0) and builds in a single command with no external quantizer, no calibration data and no patch — but ~35 % slower on MTP decode. Both are collected here.

Related work — other NInfer abliterated artifacts

Other people got here first, and their artifacts are also good. Use whichever fits — this list is here so you can find them.

Worth noting three are built on orcarouter/Qwen3.8-27B-Uncensored, which is gated.

As far as I can tell, none of the above ship a conversion recipe. That is what this repo adds.

Licence

Apache-2.0, inherited from the entire chain — every upstream component is Apache-2.0:

Apache-2.0 is not viral — it does not force a licence on you — but redistributing a derivative does carry obligations, and this repository meets them:

• LICENSE is the licence file distributed with the upstream checkpoint, retaining its Copyright 2026 Alibaba Cloud notice (§4(c)).
• NOTICE carries the attribution chain and an explicit statement of changes (§4(b)) covering both the weights and the four-line converter patch.
• No upstream component ships a NOTICE file, so §4(d) does not apply — ours is added as good practice, not obligation.

Calibration used HuggingFaceH4/ultrachat_200k (MIT). Calibration affects only the per-matrix input_global_scale scalars; no dataset content is reproduced in the artifact.

"Qwen" is a trademark of Alibaba Cloud. This is an unofficial community derivative, not endorsed by or affiliated with Alibaba Cloud, huihui-ai, unsloth, or the NInfer project.

This is a description of what the licences say, not legal advice.

Intended use and limitations

This is an uncensored model: the upstream abliteration removes the refusal direction, so it will attempt requests a stock instruction-tuned model declines. It is published for research, evaluation and local deployment where that behaviour is understood and wanted. Anyone deploying it is responsible for their own safeguards and for compliance with the Apache-2.0 licence and applicable law. The refusal behaviour, and any capability regression from the ablation, originate entirely upstream — this repository changes format, not behaviour.

Credit for the abliteration goes to @huihui-ai; for the base model to Qwen; for the engine and artifact contract to @Neroued; and for the published NVFP4 recipe to unsloth.