Qwen3.8-27B-Uncensored-NVFP4

Qwen3.8-27B-Uncensored-NVFP4 (mixed-precision, vision retained)

English | 中文

FP4-quantized build of orcarouter/Qwen3.8-27B-Uncensored (BF16), preserving the abliteration edits byte-for-byte — this is a quantization of the uncensored weights, NOT of the official Qwen3.8. The vision tower (BF16) and the full multimodal config are included, so the model works for both text-only and image/video serving.

Quantization recipe (mixed precision, mirrors nvidia/Qwen3.8-27B-NVFP4)

• Producer: NVIDIA Model Optimizer (modelopt) 0.46.0
• Calibration: cnn_dailymail (3.0.0), 512 samples × 2048 tokens, max algorithm
• KV cache: bf16/fp8 both work (no kv scales shipped — serve with default kv dtype)

Eval (vs the FP8 source build, on our own stack)

• L1 reasoning suite (semantic+arithmetic): 6/6 (source: 6/6)
• L2 contamination-resistant suite (AIME 2025 pair-counting, combinatorics, number theory, LIS coding, Chinese logic, strict format): 6/6 (source: 6/6)
• Refusal behaviour: the quantization source is the abliterated BF16 checkpoint, so the abliteration edits are preserved by construction. No refusal benchmark (e.g. AdvBench/StrongREJECT) was run by the uploader on this quantized build.

Serving

SGLang (validated, our fork of main ~2026-08-31 with modelopt MIXED support)

python -m sglang.launch_server \
  --model-path /path/to/Qwen3.8-27B-Uncensored-NVFP4-v2 \
  --language-only `# optional: text-only serving skips vision weights` \
  --tp 1

• On Ada (SM89, e.g. RTX 4090): FP4 GEMMs run via Marlin W4A16 automatically. Set SGLANG_DISABLE_SILU_FP4_QUANT_FUSION=1 (the fused SiLU+FP4 kernel has no Ada backend in current flashinfer and will crash startup with Invalid backend: 89 otherwise).
• On Blackwell: native FP4 tensor cores via flashinfer/cutlass backends.

The modelopt_fp4 (W4A4) compressed-tensors builds published elsewhere may refuse to load on Ada (min_capability=100). This checkpoint ships the W4A16-serialized form, which runs on SM89.

vLLM

MIXED_PRECISION modelopt checkpoints should load like nvidia/Qwen3.8-27B-NVFP4 (vllm ≥ 0.27). Not validated by us — feedback welcome.

Hardware notes (RTX 4090 48G, sglang)

Decode wins come from FP4 weight bandwidth + FP8 attention projections; prefill pays a Marlin dequant tax on Ada only (Blackwell has native FP4 compute). Measured on one RTX 4090 48G, TP1, DFlash2 speculative decoding K=8, fp8 KV cache.

Files

• model-0000{1,2}-of-00003.safetensors — quantized language model (20.2 GB)
• model-00003-of-00003.safetensors — BF16 vision tower, 333 tensors (0.92 GB, from the base checkpoint)
• hf_quant_config.json — modelopt MIXED_PRECISION manifest (per-layer recipe)
• LICENSE — Apache-2.0, inherited from the base checkpoint
• README.zh.md — Chinese version of this document
• full tokenizer + processor + chat template files included

Reproduction scripts (see next section):

• quantization_reproduce.py — step 1: quantize + export
• vision_merge.py — step 2: copy the BF16 vision tower in
• make_composite_config.py — step 3: rebuild config.json into the composite VL shape

Reproducing this build

# 1. quantize (mixed precision) -> writes a flat-config export
python quantization_reproduce.py \
    --src /path/to/Qwen3.8-27B-Uncensored \
    --dst /path/to/out

# 2. merge the BF16 vision tower (no requant; file-level tensor copy)
python vision_merge.py --base /path/to/Qwen3.8-27B-Uncensored --dst /path/to/out

# 3. rebuild config.json into the composite VL shape
python make_composite_config.py --base /path/to/Qwen3.8-27B-Uncensored --dst /path/to/out

Step 3 is not optional for SGLang multimodal serving. export_hf_checkpoint writes a flat text config (architectures: ["Qwen3_5ForCausalLM"]), but the encoder/decoder split whitelist used by --language-only only accepts the composite name Qwen3_5ForConditionalGeneration with a nested text_config. A flat export loads as text-only but is rejected by the split hook.

Two modelopt 0.46.0 gotchas the scripts already handle (both cost a failed run each if you hit them from scratch):

• {"algorithm": "max"} is required in mtq.quantize(...). Without it, calibrate() dispatches to NoneCalibrateMode and silently skips the forward loop — the run "succeeds" in seconds with every amax unset.
• _amax export bug. The FP8 exporter reads weight_quantizer._amax, but TensorQuantizer now exposes it as the amax property. quantization_reproduce.py wraps _export_quantized_weight to rebind it temporarily.

Requirements: torch, transformers, modelopt==0.46.0, one CUDA GPU. The BF16 weights are ~54 GB and do not fit a 48 GB card, so device_map spills to host RAM (--gpu-mem / --cpu-mem control the split; defaults 24GiB / 80GiB).

License & attribution

Apache-2.0, same as the base. Abliteration by orcarouter (see base repo); quantization pipeline by the uploader. If you redistribute, keep the base model attribution.