Qwen3.8-Flash-Next-Abliterated-NVFP4-FP8PBWO

Qwen3.8-Flash-Next — Abliterated, NVFP4 experts + FP8_PB_WO dense

An abliterated build of Qwen3.8-Flash-Next in the modelopt MIXED_PRECISION format: NVFP4 group-16 experts with FP8_PB_WO (128x128 block, weight-only) dense layers, and the PLE n-gram table in FP8.

This combination had no abliterated build. Every published one is uniform NVFP4 or compressed-tensors, both of which lose the block-scaled FP8 dense path — which on sm120 is what the KV pool depends on. This exists to fill that gap.

It is a transplant, not a new abliteration. The refusal direction was recovered from published weight deltas of an existing abliterated checkpoint and re-applied to this quantization. See METHOD.md.

Attribution

The credit for the ablation belongs upstream. What is new here is the transplant into a different quantization, and the measurement that says it worked.

What was changed

W' = W - 1.5013 * r (r^T W) applied to 24,676 residual-writing tensors:

alpha = 1.5013 was measured from the reference build (sd 0.0004 across 12 tensors), not assumed — it is a deliberate over-projection, not a plain rank-1 projection. Attention q/k/v, expert gate_proj/up_proj, norms and the vision tower are untouched.

Evaluation

GSM8K, 1319 problems, temperature 0.6, top_p 0.95, max_tokens 8192, identical harness and settings for both:

No measurable capability cost. Refusal behaviour does change on prompts the base declines. Multimodal transcription is unchanged (same image-token count, same output structure, 4.4 s vs 4.5 s).

Serving

Built and tested on SGLang, one RTX PRO 6000 Blackwell (sm120, 96 GB):

--quantization modelopt --kv-cache-dtype fp8_e4m3 --ple-offload-embedding
--mem-fraction-static 0.975 --context-length 850000
--speculative-algorithm NEXTN --speculative-num-steps 3

Serving profile is identical to the unmodified quantization: KV pool 853,056 tokens, 850k context, 4.07 GB free VRAM. --ple-offload-embedding is required — the n-gram table is ~47.7 GiB and lives in host RAM.

Note reasoning_effort matters a lot on this model. Without it, it will spend an entire token budget thinking and return an empty response; "reasoning_effort": "medium" in chat_template_kwargs fixes it.

Caveats

• The quality of the ablation is inherited from the upstream build.
• Single-direction abliteration is contested — see There Is More to Refusal in LLMs than a Single Direction.
• GSM8K is one benchmark. Unchanged arithmetic does not prove unchanged everything.
• Safety alignment has been substantially removed. It will answer things the original refuses. Released for research and for local use on hardware you own.

License follows the base model (other / Qwen terms), as with every checkpoint in the chain above.