Downloads · 30 days
6.8K
32% of all-time downloads
bjk110/Qwen3.5-122B-A10B-abliterix-NVFP4
Qwen3.5-122B-A10B-abliterix-NVFP4 is a text generation model from bjk110. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
NVFP4 (W4A4) quantization of wangzhang/Qwen3.5-122B-A10B-abliterix — the 122B/A10B Abliterix-abliterated (uncensored) Qwen3.5 MoE — packaged for vLLM serving on NVIDIA Blackwell hardware (DGX Spark GB10 / SM121).
Downloads · 30 days
6.8K
32% of all-time downloads
All-time downloads
21.3K
Public
Parameters
70.8B
75.6 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors75.5 GB · 100%
How the weights are stored.
U858.7B · 83%
From the Hugging Face model README
NVFP4 (W4A4) quantization of wangzhang/Qwen3.5-122B-A10B-abliterix — the 122B/A10B Abliterix-abliterated (uncensored) Qwen3.5 MoE — packaged for vLLM serving on NVIDIA Blackwell hardware (DGX Spark GB10 / SM121).
| Base model | wangzhang/Qwen3.5-122B-A10B-abliterix (BF16) |
| Quantization | NVFP4 W4A4 — group-wise (group_size=16), tensor-group strategy, symmetric |
| Format | compressed-tensors / nvfp4-pack-quantized (vLLM-native, packed 2 fp4/byte) |
| Weight layout (per Linear) | .weight_packed (uint8, packed) + .weight_scale (fp8 e4m3fn, per-group) + .weight_global_scale (float32) + .input_global_scale (float32) |
| Activation quant | dynamic local, group_size=16, fp8 e4m3fn scale |
| Skipped modules | lm_head, *.mlp.gate, *.mlp.shared_expert_gate, *.embed_tokens, all *.linear_attn.* (Mamba/GDN), all visual.* (text-only model) — kept BF16 |
| NVFP4 layers | 37,056 |
| BF16 layers preserved | 471 (norms, biases, A_log, dt_bias, embed_tokens, lm_head, gates, linear_attn) |
| Shards | 15 × ~5.4 GB safetensors |
| Total size on disk | 70.4 GB (~32% of BF16 baseline) |
| Tested vLLM image | ghcr.io/bjk110/vllm-spark:v022-d568 |
| Runtime stack | NGC pytorch:26.04-py3 base • PyTorch 2.12.0a0 • CUDA 13.0 • vLLM v0.21.0 + PR #35568 cherry-pick • FlashInfer v0.6.11.post3 • NCCL 2.30.4 • Triton 3.7.0 • TensorRT 5.8.1 |
| Topology | 2× DGX Spark GB10, TP=2 over 200 Gbps RoCE |
The Abliterix-trained uncensored behavior (0.5% refusal rate, KL divergence 0.0115 vs the Qwen3.5-122B-A10B baseline) is preserved while dropping weight memory from BF16 (~230 GB) to NVFP4 (~70 GB) — fitting comfortably on two DGX Spark nodes (2 × 119 GiB unified memory) with headroom for KV-cache, with the same compressed-tensors runtime path as the FP8 sibling.
Compared to the FP8 W8A8 variant (bjk110/Qwen3.5-122B-A10B-abliterix-FP8, 116 GB), this NVFP4 build cuts another ~40% off-disk and unlocks the FlashInfer-CUTLASS NVFP4 MoE backend for higher throughput on Blackwell.
Direct safetensors-level conversion via convert_bf16_to_nvfp4.py, mirroring the layout of RedHatAI/Qwen3.5-122B-A10B-NVFP4 (built with llm-compressor) — but applied directly to the abliterix BF16 checkpoint, because llm-compressor 0.10 pins transformers <=4.57.6 while Qwen3.5MoeForCausalLM is only available in transformers >=5.5. The direct script needs only torch + safetensors.
For each 2D Linear weight W (shape [out, in]) not in the ignore list:
q_proj, k_proj, v_proj of the same self_attn → 3-member groupexperts.N.{gate_proj, up_proj} → 2-member groupshared_expert.{gate_proj, up_proj} → 2-member groupo_proj, down_proj) is independent.weight_global_scale (float32). For a fused group, all members share one global scale computed from max(abs(W)) across the entire group; vLLM merges these Linears at runtime (fused QKV / gate_up matmul) and requires a single coherent scale per merged matrix — independent per-projection scales produce garbage outputs (repeated tokens) even though the model boots.W into groups of 16 along the input dim. Compute per-group fp8 e4m3fn weight_scale (one scale per 16-element group) so that each group fits the NVFP4 [-6, 6] range after dequant.e2m1), pack two values per uint8 → weight_packed.input_global_scale (float32, per-tensor) is stored for the runtime to compute on-the-fly NVFP4 activation quantization.Activations are NVFP4 W4A4 — quantized at inference time by vLLM using the stored input_global_scale with dynamic local group_size=16 (fp8 e4m3fn scale).
config.json quantization block (excerpt){
"quantization_config": {
"config_groups": {
"group_0": {
"format": "nvfp4-pack-quantized",
"targets": ["Linear"],
"weights": {
"num_bits": 4, "type": "float", "strategy": "tensor_group",
"group_size": 16, "symmetric": true, "dynamic": false,
"observer": "memoryless_minmax",
"scale_dtype": "torch.float8_e4m3fn"
},
"input_activations": {
"num_bits": 4, "type": "float", "strategy": "tensor_group",
"group_size": 16, "symmetric": true, "dynamic": "local",
"observer": "static_minmax",
"scale_dtype": "torch.float8_e4m3fn"
},
"output_activations": null
}
},
"format": "nvfp4-pack-quantized",
"ignore": [
"re:.*lm_head",
"re:visual.*",
"re:model.visual.*",
"re:.*mlp.gate$",
"re:.*embed_tokens$",
"re:.*shared_expert_gate$",
"re:.*linear_attn.*"
],
"quant_method": "compressed-tensors"
}
}
# Tested with the v022-d568 image — see DGX Spark notes below.
vllm serve bjk110/Qwen3.5-122B-A10B-abliterix-NVFP4 \
--tensor-parallel-size 2 \
--distributed-executor-backend ray \
--quantization compressed-tensors \
--moe-backend flashinfer_cutlass \
--kv-cache-dtype fp8 \
--max-model-len 32768 \
--max-num-seqs 4 \
--gpu-memory-utilization 0.88 \
--enable-chunked-prefill \
--reasoning-parser qwen3
--moe-backend flashinfer_cutlass selects the FlashInfer CUTLASS NVFP4 grouped-GEMM kernel for the MoE expert path — required to get the full NVFP4 speedup on Blackwell.
A single Blackwell-class GPU is not enough to hold the 70 GB weights + KV cache + activation buffers, so --tensor-parallel-size 2 (or higher) is mandatory.
NVIDIA DGX Spark uses SM121, which on stock vLLM v0.21.0 was excluded from the Marlin/CUTLASS NVFP4 codepaths (the gates were SM120-only). vLLM PR #35568 (commit 06d020bb6) widens those gates to the SM12x family — covering both the FP8 W8A8 and the NVFP4 dispatch paths.
With that fix applied, the boot log reports the NVFP4 kernels are bound to the SM121 GPUs and the FlashInfer-CUTLASS MoE path is selected, confirming the full Blackwell NVFP4 pipeline is active.
v022-d568)The image is the cumulative top of the v022 forward-stack build chain, rooted in NGC nvcr.io/nvidia/pytorch:26.04-py3 (CUDA 13.0, PyTorch 2.12.0a0). Each layer corresponds to one published image tag:
| Stack layer | Component / version | Image tag |
|---|---|---|
| Base | NGC pytorch:26.04-py3 (CUDA 13.0, PyTorch 2.12.0a0) | v022-ngc2604 |
| Inference | vLLM v0.21.0 | v022-vllm021 |
| FP4/FP8 attention & MoE kernels | FlashInfer v0.6.11.post3 | v022-fi0611 |
| Triton | 3.7.0 | v022-trt37 |
| TensorRT runtime | 5.8.1 | v022-tx581 |
| Collective comm | NCCL 2.30.4 | v022-nccl234 |
| SM121 enablement | vLLM PR #35568 cherry-pick (SM120 → SM12x gates) | v022-d568 ← this |
Building on NGC 26.04 (vs. the older 26.03 base used by v021) gives the SM121 GPU the matching CUDA 13.0 driver/runtime split that the Blackwell NVFP4 kernels expect, and is required for FlashInfer v0.6.11.post3 (which assumes CUDA 13 headers).
| Stage | Repo / Tag |
|---|---|
| BF16 baseline | Qwen/Qwen3.5-122B-A10B |
| BF16 reference NVFP4 (text+vision) | RedHatAI/Qwen3.5-122B-A10B-NVFP4 |
| Abliterix abliteration (BF16) | wangzhang/Qwen3.5-122B-A10B-abliterix |
| FP8 W8A8 abliterix | bjk110/Qwen3.5-122B-A10B-abliterix-FP8 |
| NVFP4 W4A4 (this repo) | bjk110/Qwen3.5-122B-A10B-abliterix-NVFP4 |
@software{abliterix,
author = {Wu, Wangzhang},
title = {Abliterix: Automated LLM Abliteration},
year = {2026},
url = {https://github.com/wuwangzhang1216/abliterix}
}
compressed-tensors + FlashInfer-CUTLASS NVFP4 MoE integration teams.