Qwen3.5-122B-A7B-Int4

Qwen3.5-122B-A7B-Int4

A top-k=4 pruned and healed INT4 build of Qwen3.5-122B-A10B, with a retrained MTP draft head, tuned for single-node inference on a DGX Spark.

⚠️ Read this first — you need the hybrid vLLM patches

This checkpoint keeps its shared expert in BF16 while everything else is INT4. Stock vLLM sends any module with bits >= 16 to UnquantizedLinearMethod, which drops those 144 projections off the fused quantized path. It still runs — it just runs ~27% slower.

Measured on identical hardware, same model, same vLLM version (0.26.1rc1.dev1123), patches on vs off:

Get the patches here:

https://github.com/azampatti/vllm-hybrid-int4-fp8-patches

They apply on top of an existing vLLM image and leave the base image untouched.

---

Speed

🚀 70.8 tok/s on long-form code generation

Full bench.sh sweep, single stream, DGX Spark, patched vLLM, MTP depth 3:

Caveat on all speed numbers. These are DGX Spark (GB10 / SM121) figures. Everything here was built and tested only on a DGX Spark, on a vLLM image derived from eugr/spark-vllm-docker. No other hardware or base image has been validated. Your numbers will differ.

---

What was done

1. Top-k=4 expert pruning

The base model routes each token to 8 of its 256 experts. This build routes to 4, halving routed-expert compute and weight traffic per token. Nothing is deleted — all 256 experts are still present and the router still chooses among them; it simply selects half as many per token.

Done naively this costs real accuracy: the model was trained expecting eight opinions per token and now gets four.

2. Shared-expert healing

Every layer also has a shared expert that is always active, regardless of routing. That module is the natural place to absorb what the four dropped experts were contributing, so it was retrained in BF16 by distillation against the unpruned base model while the rest of the network stayed frozen.

The shared expert is deliberately left unquantized (BF16). Quantizing it was measured and rejected — it is the one module carrying the trained correction, and quantization noise on it is comparable in size to the correction itself. That choice is exactly why the hybrid patches above are required for full speed.

After healing, quality matches the original base model.

3. MTP draft-head retraining

The multi-token-prediction head used for speculative decoding is itself a 256-expert MoE layer, so cutting the target's top-k cut the draft head with it. The head was retrained with KL distillation against the target model and the best checkpoint is shipped here. Speculative decoding is distribution-preserving — it changes speed, never output quality.

Serve with num_speculative_tokens: 3 (depth 3); the launch command below does.

---

Model details

The A7B in the name follows the base model's A10B convention. Counted directly from config.json, this build activates ~6.4 B parameters per token including embeddings and the LM head, or ~4.9 B across the 48 transformer layers alone.

---

Launching

Replace <hybrid-image> with the tag you built from the patch repo. The model is pulled from the Hugging Face cache — no local path needed.

docker run --privileged --gpus all -d \
  --name vllm-qwen35-a7b \
  --net=host \
  --ipc=host \
  -v "${HOME}/.cache/huggingface:/root/.cache/huggingface" \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e VLLM_MTP_TOP_K=8 \
  <hybrid-image> \
  vllm serve azampatti/Qwen3.5-122B-A7B-Int4 \
    --served-model-name Qwen3.5-122B-A7B-Int4 \
    --port 8000 \
    --host 0.0.0.0 \
    --speculative-config '{"method":"mtp","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN"}' \
    --max-model-len 200K \
    --gpu-memory-utilization 0.75 \
    --load-format fastsafetensors \
    --attention-backend TRITON_ATTN \
    --dtype bfloat16 \
    --kv-cache-dtype fp8 \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --max-num-seqs 8 \
    --max-num-batched-tokens 16384 \
    --override-generation-config '{"temperature":0.5,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

Then:

curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen3.5-122B-A7B-Int4",
       "messages":[{"role":"user","content":"Write a binary search in Rust."}]}'

Notes on the flags

• VLLM_MTP_TOP_K=8 is provided by the patches. The MTP draft head is a 256-expert MoE layer in its own right, and this lets it route more widely than the top-4 target. It affects speed only.
• The chat template ships with the model (chat_template.jinja) and is loaded automatically. It carries the tool-calling format this build expects — pass --chat-template only if you intend to override it.
• --reasoning-parser qwen3 and --tool-call-parser qwen3_coder are required for thinking blocks and tool calls to be parsed into their own response fields rather than appearing as raw text.
• --kv-cache-dtype fp8 and --gpu-memory-utilization 0.75 are sized for a 128 GB DGX Spark at 200K context. Raise the utilization if you have headroom.

---

License

Apache 2.0, inherited from Qwen3.5-122B-A10B.