qwen3.6-35b-abliterated-nvfp4-modelopt

Qwen3.6-35B-A3B Abliterated NVFP4 (ModelOpt)

Abliterated and NVFP4-quantized version of Qwen/Qwen3.6-35B-A3B. Optimized for NVIDIA Blackwell (DGX Spark / GB10).

Performance

2x NVIDIA DGX Spark (GB10, SM121 Blackwell) — TP=2, CUDA Graphs

Single DGX Spark — CUDA Graphs

What's in this model?

1. Abliteration: Refusal directions removed from layers 14-26 using norm-preserving biprojected abliteration
2. NVFP4 Quantization: Quantized with NVIDIA ModelOpt (v0.43.0) using 256 calibration samples from Open-Platypus, batch_size=64
3. Format: ModelOpt FP4 — compatible with vLLM (--quantization modelopt_fp4)

Model Details

• Base Model: Qwen/Qwen3.6-35B-A3B (Apache 2.0)
• Architecture: MoE, 35B total params, 3B active per token, 256 experts (8 routed + 1 shared)
• Quantization: NVFP4 on language model linear layers. Vision encoder, gates, conv1d, lm_head excluded.
• Size: ~20GB (vs 67GB BF16, vs 35GB FP8)
• Context: 32K (configurable)

Usage with vLLM

Single Node

vllm serve joshebbs/qwen3.6-35b-abliterated-nvfp4-modelopt \
    --quantization modelopt_fp4 \
    --trust-remote-code \
    --dtype auto \
    --kv-cache-dtype fp8 \
    --attention-backend flashinfer \
    -tp 1

Multi-Node (2x DGX Spark)

vllm serve joshebbs/qwen3.6-35b-abliterated-nvfp4-modelopt \
    --quantization modelopt_fp4 \
    --trust-remote-code \
    --dtype auto \
    --kv-cache-dtype fp8 \
    --attention-backend flashinfer \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    -tp 2 --nnodes 2 --node-rank 0 \
    --master-addr <HEAD_IP> --master-port 29501

Important Notes

• Vision: This checkpoint includes the ConditionalGeneration config for compatibility, but vision weights are not included. The visual encoder dimensions (4304) are not NVFP4-aligned for TP>1. For vision support, use the base FP8 model.
• First startup: FlashInfer JIT compilation takes ~25-30 minutes on first run. Subsequent starts use cached kernels (~5-7 minutes).
• Multi-node timeout: The mp distributed backend requires patching store_timeout in vllm/distributed/utils.py from 300s to 1800s for first-run JIT compilation across nodes.
• FLASHINFER_NVCC_THREADS: Set to 16 for faster JIT compilation on multi-core systems.

Quantization Recipe

Quantized on NVIDIA H200 via Vast.ai in 7.4 minutes ($3.31):

import modelopt.torch.quantization as mtq
quant_config = mtq.NVFP4_DEFAULT_CFG
model = mtq.quantize(model, quant_config, forward_loop=calibrate_loop)
# calibrate_loop: 256 samples, batch_size=64, max_seq_length=1024

Abliteration Details

• Tool: jim-plus/llm-abliteration
• Measurement: Projected mode with flash attention on BF16 model
• Layers: 14-26 (strongest refusal signal, signal quality 0.14-0.22)
• Scale: 1.0, Sparsity: 0.0

Hardware Tested

• 2x NVIDIA DGX Spark (GB10 Blackwell, 128GB unified memory, 200Gb/s QSFP RoCE)
• NVIDIA H200 (144GB HBM3e) — validation only