Qwen3.5-397B-A17B-heretic-int4-AutoRound

Qwen3.5-397B-A17B-heretic-int4-AutoRound

The first INT4 AutoRound quantization of the Heretic (uncensored) Qwen3.5-397B — the largest open-source abliterated model.

A 4-bit symmetric quantization of trohrbaugh/Qwen3.5-397B-A17B-heretic, generated using Intel AutoRound v0.12.0. The original multimodal architecture (Qwen3_5MoeForConditionalGeneration) is fully preserved — text, vision, video, and reasoning all work.

---

Model Lineage

This model has a three-step provenance chain:

Qwen/Qwen3.5-397B-A17B            ← Qwen team's 397B MoE frontier model (BF16, ~800GB)
  └─► trohrbaugh/heretic           ← Abliterated with Heretic v1.2.0
       └─► THIS MODEL              ← INT4 AutoRound quantization (198GB, vision preserved)

1. Qwen3.5-397B-A17B — Qwen team's largest Mixture-of-Experts model with Gated DeltaNet hybrid attention, 512 experts (16 active + 1 shared per token), native 262K context, and multimodal vision support.

2. trohrbaugh/Qwen3.5-397B-A17B-heretic — Abliterated (uncensored) variant using Heretic by p-e-w. Uses parametrized directional ablation applied to attn.o_proj and mlp.down_proj across all layers.

3. This model — INT4 AutoRound quantization targeting unified-memory GPU systems (NVIDIA DGX Spark / ASUS Ascent GX10). All vision weights, shared expert layers, MoE gate layers, and lm_head preserved at full precision. Text module quantized to INT4 W4G128.

---

Quantization Details

What Was Preserved (NOT Quantized)

Keeping critical layers at full precision is essential for MoE model quality:

• Vision encoder weights (model.visual.blocks.*) — BF16
• Shared expert projections (layers × gate_proj, up_proj, down_proj) — FP16
• Shared expert gates (shared_expert_gate) — FP16
• MoE routing gates — FP16
• lm_head — original precision

The shared expert is activated for every token (unlike the 511 routed experts where only 16 fire per token). Preserving its precision is critical for maintaining output quality.

---

Model Size Comparison

The 13 GB difference from Intel's canonical INT4 is because the Heretic fork does not include MTP (Multi-Token Prediction) weights. MTP is a speculative decoding feature — its absence does not affect output quality.

---

Performance (Measured on DGX Spark Cluster)

Tested on a 2-node NVIDIA DGX Spark cluster (2× Grace Blackwell GB10, 128GB unified memory each) with tensor parallelism over 200Gbps RoCE RDMA. Deployment configuration heavily informed by the spark-vllm-docker community project and the NVIDIA DGX Spark Developer Forum.

Speed: 43% Faster Than Canonical

The Heretic is significantly faster than the canonical quantization on identical hardware. The speed gain likely comes from the smaller memory footprint (no MTP weights) reducing memory pressure.

Note: TTFT is higher on the Heretic (5.6s vs 0.87s) because this was measured during a thinking-disabled coding evaluation with a longer system prompt. Single-request TTFT with short prompts is comparable.

---

Coding Benchmark — Full-Stack Evaluation

We ran an identical full-stack coding evaluation across all models: "Build a task manager with Bun, Hono, React, PostgreSQL, Drizzle ORM."

Results

Key Finding: Do NOT Use Thinking Mode

Thinking mode HURTS this model's code quality:

This is the opposite of the canonical model, where thinking helps (+4%). The abliteration appears to weaken the reasoning-to-execution pipeline — the model's thinking layer notes best practices (e.g., "usually use bcrypt") but then doesn't implement them.

Recommendation: Always use "chat_template_kwargs": {"enable_thinking": false} with this model for code generation.

Without Thinking: Close to Canonical Quality

When thinking is disabled, the quality gap is small:

The Heretic actually implements better auth security (bcrypt + httpOnly + ownership checks) — the canonical model used jose JWT without password hashing. The Heretic's extra hard blockers are mostly import/config issues rather than architectural flaws.

---

Quick Start

vLLM (Recommended)

For a single multi-GPU server with enough VRAM (e.g., 2×H100 80GB, 4×A100 80GB):

pip install vllm>=0.17.0

vllm serve happypatrick/Qwen3.5-397B-A17B-heretic-int4-AutoRound \
  --served-model-name qwen3.5-397b \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 2 \
  --max-model-len 65536 \
  --kv-cache-dtype fp8 \
  --language-model-only \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --chat-template-kwargs '{"enable_thinking": false}' \
  --trust-remote-code

Important: Use --language-model-only to skip loading the vision encoder if you only need text/code. Adjust --tensor-parallel-size and --max-model-len for your hardware.

Then query with any OpenAI-compatible client:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5-397b",
    "messages": [{"role": "user", "content": "Write a Python quicksort implementation"}],
    "temperature": 0.6
  }'

HuggingFace Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "happypatrick/Qwen3.5-397B-A17B-heretic-int4-AutoRound"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    trust_remote_code=True,
)

messages = [{"role": "user", "content": "Hello!"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

---

DGX Spark / ASUS Ascent GX10 Deployment Guide

This model requires a dual-node DGX Spark cluster (2× 128GB unified memory). It does NOT fit on a single DGX Spark.

Our deployment relies heavily on the spark-vllm-docker community project by eugr, which provides pre-built vLLM + FlashInfer wheels for SM12.1a (the GB10's compute capability). Additional deployment insights came from JungkwanBan's archived recipe (now merged into spark-vllm-docker) and the NVIDIA DGX Spark Developer Forum.

Dual Node (2× DGX Spark, 256 GB Total)

For two DGX Sparks connected via 200Gbps RoCE RDMA, using native distributed mode (no Ray — lower memory overhead):

# Node 2 (worker, rank 1) — start first, waits for head
docker run -d --name vllm-worker \
  --gpus all --network=host --ipc=host --privileged \
  --shm-size 10.24g \
  -v /path/to/models:/models \
  -e NCCL_SOCKET_IFNAME=enP2p1s0f0np0 \
  -e NCCL_IB_HCA=roceP2p1s0f0 \
  -e NCCL_P2P_DISABLE=1 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e VLLM_MARLIN_USE_ATOMIC_ADD=1 \
  -e OMP_NUM_THREADS=4 -e MKL_NUM_THREADS=4 -e TORCH_NUM_THREADS=4 \
  <vllm-image> \
  bash -c "vllm serve /models/heretic-397b-int4-autoround \
    --host 0.0.0.0 --port 8000 \
    --tensor-parallel-size 2 \
    --max-model-len 262144 \
    --max-num-seqs 2 \
    --max-num-batched-tokens 4176 \
    --gpu-memory-utilization-gb 112 \
    --kv-cache-dtype fp8 \
    --enable-prefix-caching \
    --trust-remote-code \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --reasoning-parser qwen3 \
    --chat-template unsloth.jinja \
    --nnodes 2 --node-rank 1 \
    --master-addr <NODE1_IP> --master-port 29500 \
    --headless"

sleep 5

# Node 1 (head, rank 0) — triggers model loading
docker run -d --name vllm-head \
  --gpus all --network=host --ipc=host --privileged \
  --shm-size 10.24g \
  -v /path/to/models:/models \
  -e NCCL_SOCKET_IFNAME=enP2p1s0f0np0 \
  -e NCCL_IB_HCA=roceP2p1s0f0 \
  -e NCCL_P2P_DISABLE=1 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e VLLM_MARLIN_USE_ATOMIC_ADD=1 \
  -e OMP_NUM_THREADS=4 -e MKL_NUM_THREADS=4 -e TORCH_NUM_THREADS=4 \
  <vllm-image> \
  bash -c "vllm serve /models/heretic-397b-int4-autoround \
    --served-model-name qwen3.5-397b \
    --host 0.0.0.0 --port 8000 \
    --tensor-parallel-size 2 \
    --max-model-len 262144 \
    --max-num-seqs 2 \
    --max-num-batched-tokens 4176 \
    --gpu-memory-utilization-gb 112 \
    --kv-cache-dtype fp8 \
    --enable-prefix-caching \
    --trust-remote-code \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --reasoning-parser qwen3 \
    --chat-template unsloth.jinja \
    --nnodes 2 --node-rank 0 \
    --master-addr <NODE1_IP> --master-port 29500"

Note on <vllm-image>: Standard vllm/vllm-openai does not ship FlashInfer kernels for SM12.1a (GB10). Use spark-vllm-docker vllm-node-tf5 image or build your own. Our tested configuration uses vLLM 0.18.1rc1.dev69 + FlashInfer 0.6.7.

DGX Spark Tips & Gotchas

These are hard-won lessons from our deployment and the community:

• --gpu-memory-utilization-gb 112 — absolute GiB allocation, NOT a percentage. This is the maximum for ASUS Ascent UMA systems. Requires patching vLLM's request_memory() safety margin check from raise ValueError to logging.warning — cudaMemGetInfo.free underreports by 3–4 GiB because reclaimable page cache is counted as "used"
• --shm-size 10.24g is critical for multi-node — without it, model loading can hang at ~57% (NCCL deadlock)
• Do NOT use --enforce-eager — CUDA graphs on GB10 with driver 580 actually save 0.21 GiB of memory. The myth that --enforce-eager is required for native distributed is false (confirmed by JungkwanBan's testing and our own)
• NCCL_P2P_DISABLE=1 is required for GB10's unified memory architecture — P2P transfers hang
• --kv-cache-dtype fp8 is essential — the 397B model needs all available memory, fp8 halves KV cache size. Results in 9.41 GiB for 267,264 tokens capacity
• --language-model-only saves ~3GB by not loading the vision encoder (optional — omit if you need vision)
• --enable-prefix-caching — free memory savings via KV cache reuse for shared prefixes
• Disable the desktop GUI (sudo systemctl set-default multi-user.target) — the GNOME compositor contends with Marlin weight repacking and wastes 400–500 MiB per node. See the NVIDIA forum thread
• Set vm.swappiness=10 at the OS level — reduces swapping of GPU-mapped pages
• Thread-limiting env vars (OMP_NUM_THREADS=4, MKL_NUM_THREADS=4, TORCH_NUM_THREADS=4) — community-recommended for GB10 to avoid oversubscribing the Grace CPU cores during model loading
• Drop page caches before launching vllm serve — sync; echo 3 > /proc/sys/vm/drop_caches inside both containers recovers reclaimable memory
• Model startup takes ~16.5 minutes cold (weight loading + torch.compile + FlashInfer autotuning + CUDA graph capture). ~8 minutes with AOT cache on subsequent launches

Post-Quantization Fixes Applied

This model required several fixes after the initial AutoRound quantization to work correctly with vLLM:

1. config.json — Fixed block_name_to_quantize from ["model.language_model.layers", "model.visual.blocks"] to "model.language_model.layers" (string, not array). The original told vLLM to create quantized Marlin layers for the vision encoder, causing a MarlinLinearKernel crash because the vision embed_dim (1152) ÷ TP(2) = 576, which is not divisible by min_thread_k (128). Also fixed extra_config key prefix from "" to "model.language_model."
2. quantization_config.json — Replaced with Intel's reference file (the AutoRound output was incomplete)
3. Shard 39 — Merged 333 visual encoder keys from model_extra_tensors.safetensors into model-00039-of-00040.safetensors (vLLM ignores extra tensors files)
4. processor_config.json — Added (required for multimodal model loading)
5. model.safetensors.index.json — Updated weight map to reflect the merged shard and removed references to model_extra_tensors.safetensors

These fixes are documented in detail in our 397B deployment log.

---

Abliteration Notice

This model is based on a "Heretic" variant which has had safety refusals significantly reduced using directional ablation (Heretic by p-e-w).

This model will follow instructions that the original Qwen model would refuse. Please use responsibly. The creators are not responsible for misuse.

---

Ethical Considerations and Limitations

• This is an uncensored model. It has reduced safety guardrails compared to the original Qwen3.5-397B-A17B.
• Quantization to INT4 introduces a small quality degradation compared to BF16, though our testing shows close-to-canonical performance on code generation tasks (6.55 vs 6.85/10).
• Do not use thinking mode for code generation. It significantly degrades output quality on this model (4.80/10 with thinking vs 6.55/10 without).
• The model may generate biased, incorrect, or harmful content. Users should implement appropriate safety measures.

---

License

This model inherits the Apache 2.0 license from Qwen/Qwen3.5-397B-A17B.

---

Acknowledgments

• Qwen Team for the incredible Qwen3.5-397B-A17B base model
• trohrbaugh for the Heretic abliteration
• p-e-w for the Heretic abliteration tool
• Intel for AutoRound and the canonical quantization recipe
• eugr for the spark-vllm-docker community project — pre-built vLLM + FlashInfer wheels for SM12.1a and tested deployment recipes
• JungkwanBan for proving CUDA graphs work on GB10 with driver 580 (saves memory vs eager mode)
• NVIDIA DGX Spark Developer Forum — community knowledge on UMA memory management, GUI deadlocks, and multi-node deployment
• NVIDIA for the DGX Spark platform

---

Citation

@article{cheng2024optimize,
  title={Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs},
  author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi},
  journal={arXiv preprint arXiv:2309.05516},
  year={2024}
}

@misc{qwen3.5,
  title={Qwen3.5 Technical Report},
  author={Qwen Team},
  year={2025},
  url={https://qwenlm.github.io/blog/qwen3.5/}
}

---

Quantized by trohrbaugh. Post-quantization fixes and benchmarks by happypatrick. Tested on a 2-node DGX Spark cluster with 200Gbps RoCE RDMA. Deployment informed by the spark-vllm-docker community project and the NVIDIA DGX Spark Developer Forum.