Huihui-Qwen3.6-27B-abliterated-NVFP4

Huihui-Qwen3.6-27B-abliterated-NVFP4

English | 繁體中文

---

English

[!TIP] Quantized using llm-compressor with mixed-domain calibration and sensitive-layer protection for maximum accuracy recovery.

[!IMPORTANT] Native W4A4 on DGX Spark (SM121) — confirmed working

This checkpoint runs true W4A4 via FlashInfer CUTLASS NVFP4 kernel (verified in vLLM logs: FlashInferCutlassNvFp4LinearKernel). Requires:

• vLLM 0.19.1rc1.dev374+g1174723eb or later (includes PR #37725 arch-suffix fix)
• FlashInfer ≥ 0.6.8 with SM120f compilation (PR #2650)
• CUDA ≥ 12.9

MTP speculative decoding supported — use method: qwen3_next_mtp (dense-model MTP, distinct from the MoE mtp method).

NVFP4 W4A4 quantization of huihui-ai/Huihui-Qwen3.6-27B-abliterated, optimized for NVIDIA DGX Spark (GB10 SM121) with the FlashInfer CUTLASS FP4 kernel.

Model Details

Quantization Details

This model uses a three-strategy stack (ACD) on top of the llm-compressor official flow:

B (last-layer protection) incompatible with vLLM fused kernels: Partial ignore of linear layers within a transformer block triggers ValueError: All projections need to have same quantization scheme but found multiple.

E (SpinQuant R1+R2) incompatible with multi-modal config: llm-compressor's get_head_dim only reads top-level config, not Qwen3.6's nested text_config.

Layers Preserved in BF16

Speculative Decoding

This model supports two speculative decoding methods:

MTP (built-in weights, recommended default for this abliterated variant — see warning at top):

--speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}'

DFlash (separate drafter, recommended for single-user / low-concurrency on non-abliterated variants):

--speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-27B-DFlash", "num_speculative_tokens": 15}'

Note: On hybrid GDN architectures, MTP may hit a state-rollback bug (vLLM #39273) during high token-rejection rates. Reduce num_speculative_tokens to 1 if you observe degraded output quality.

Serving with vLLM

vllm serve /path/to/model \
    --quantization compressed-tensors \
    --served-model-name qwen3.6-27b \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --attention-backend flash_attn \
    --kv-cache-dtype auto \
    --gpu-memory-utilization 0.80 \
    --max-model-len 131072 \
    --max-num-batched-tokens 16384 \
    --max-num-seqs 8 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --performance-mode throughput \
    --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}' \
    --trust-remote-code \
    --language-model-only

DGX Spark (SM121) Compatibility Notes

• Native W4A4 confirmed via FlashInfer CUTLASS NVFP4 kernel
• Verify in logs: Using FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM
• FP8 KV cache is not compatible with GDN non-causal attention; use --kv-cache-dtype auto
• Dense model fits more comfortably on 128 GB UMA than the 35B BF16 weight — higher max-model-len is achievable
• --language-model-only skips vision encoder profiling for text-only inference
• Clear page cache before starting on UMA: sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'

Known Limitations

• Per-tensor global scales for fused q_proj / k_proj / v_proj may differ, causing a vLLM warning at load time. Impact on accuracy is typically small but measurable on strict tool-calling JSON schemas.
• DFlash drafter was trained on the original Qwen3.6-27B, not the abliterated variant — acceptance rate may be lower than on the original model.

Safety Warning

This model has safety filtering removed (abliterated) and may generate sensitive, controversial, or inappropriate content. Users are solely responsible for all consequences arising from its use. Please ensure usage complies with local laws and ethical standards. Not suitable for public-facing or production applications.

Credits

• Original Model: Qwen/Qwen3.6-27B by Alibaba Qwen Team
• Abliteration: huihui-ai
• NVFP4 Quantization: YuYu1015 on NVIDIA DGX Spark (GB10)
• Quantization Tool: llm-compressor by vLLM Project
• Reference Quantization: sakamakismile/Qwen3.6-27B-NVFP4
• Sensitivity Analysis: Diagnosing FP4 Inference (arXiv 2603.08747)

---

繁體中文

[!TIP] 使用 llm-compressor 搭配混合領域校準與敏感層保護量化，最大化精度保留。

[!IMPORTANT] DGX Spark (SM121) 原生 W4A4 — 已驗證可用

此 checkpoint 透過 FlashInfer CUTLASS NVFP4 kernel 跑真 W4A4（vLLM log 可見 FlashInferCutlassNvFp4LinearKernel）。需要：

• vLLM 0.19.1rc1.dev374+g1174723eb 以上（含 PR #37725 arch-suffix 修復）
• FlashInfer ≥ 0.6.8 帶 SM120f 編譯（PR #2650）
• CUDA ≥ 12.9

支援 MTP 投機解碼 — 使用 method: qwen3_next_mtp（dense 模型專用，有別於 MoE 的 mtp 方法）。

huihui-ai/Huihui-Qwen3.6-27B-abliterated 的 NVFP4 W4A4 量化版，針對 NVIDIA DGX Spark (GB10 SM121) 最佳化，使用 FlashInfer CUTLASS FP4 kernel。

模型資訊

量化詳情

此模型在 llm-compressor 官方流程上堆疊三項策略（ACD）：

B 策略（最後層保護）與 vLLM fused kernel 不相容：對 transformer block 內的 linear 層做 partial ignore 會觸發 ValueError: All projections need to have same quantization scheme but found multiple。

E 策略（SpinQuant R1+R2）與 multi-modal config 不相容：llm-compressor 的 get_head_dim 只讀頂層 config，不讀 Qwen3.6 巢狀的 text_config。

保留 BF16 的層

投機解碼

本模型支援兩種投機解碼方式：

MTP（內建權重，此 abliterated 變體的建議預設 — 詳見頂部警告）：

--speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}'

DFlash（獨立 drafter，建議於非 abliterated 版本 / 單用戶 / 低併發使用）：

--speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-27B-DFlash", "num_speculative_tokens": 15}'

注意：混合 GDN 架構下 MTP 可能觸發 state-rollback bug（vLLM #39273），高 rejection rate 時輸出可能退化。若遇輸出品質下降，請將 num_speculative_tokens 降為 1。

vLLM 部署

vllm serve /path/to/model \
    --quantization compressed-tensors \
    --served-model-name qwen3.6-27b \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --attention-backend flash_attn \
    --kv-cache-dtype auto \
    --gpu-memory-utilization 0.80 \
    --max-model-len 131072 \
    --max-num-batched-tokens 16384 \
    --max-num-seqs 8 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --performance-mode throughput \
    --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}' \
    --trust-remote-code \
    --language-model-only

DGX Spark (SM121) 相容性說明

• 原生 W4A4 已確認 透過 FlashInfer CUTLASS NVFP4 kernel
• log 驗證：Using FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM
• FP8 KV cache 與 GDN non-causal attention 不相容，請使用 --kv-cache-dtype auto
• Dense 模型比 35B BF16 更小，128 GB UMA 上可支援更大的 max-model-len
• --language-model-only 跳過視覺編碼器 profiling，加速純文字推理啟動
• UMA 架構啟動前請先清除 page cache：sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'

已知限制

• Fused q_proj / k_proj / v_proj 的 per-tensor global scale 可能不一致，vLLM 載入時會印警告。一般精度影響輕微，但在嚴格 tool-calling JSON schema 下可能可測得。
• DFlash drafter 是以原版 Qwen3.6-27B 訓練，非 abliterated 變體 — 接受率可能較原版低。

安全警告

此模型已移除安全過濾機制（abliterated），可能產生敏感、爭議性或不當內容。使用者須自行承擔所有風險與法律責任，並確保使用方式符合當地法規與倫理標準。不適用於公開或生產環境。

致謝

• 原始模型：Qwen/Qwen3.6-27B，Alibaba Qwen 團隊
• 去審查：huihui-ai
• NVFP4 量化：YuYu1015，於 NVIDIA DGX Spark (GB10) 上完成
• 量化工具：llm-compressor，vLLM Project
• 參考量化：sakamakismile/Qwen3.6-27B-NVFP4
• 敏感度分析：Diagnosing FP4 Inference (arXiv 2603.08747)