Qwen3.5-4B-AWQ-vd

Qwen3.5-4B-AWQ-vd

An independently re-quantized AWQ INT4 build of Qwen/Qwen3.5-4B with all linear projections quantized — self-attention (8 full-attention layers), the gated delta-net / linear-attention projections (24 layers), and the MLPs — while keeping in_proj_a/in_proj_b, norms, embeddings and the MTP head in fp16.

~35% smaller and ~35% fewer weight bytes per token than the reference QuantTrio/Qwen3.5-4B-AWQ (3.8 GB vs 5.7 GB), with output quality verified equivalent in side-by-side A/B testing.

Why this exists

The reference checkpoint leaves all attention weights in fp16. Quantizing them naively produces broken output — we root-caused two independent defects in the process:

1. The AutoAWQ fork's activation smoothing stores LN folds assuming a Llama-style norm gain (w), but Qwen3.5's norm gain is (1+w) — every fold was silently wrong at runtime while passing per-tensor checks.
2. Plain RTN/clipped quantization of the MLP layers loses ~3× more fidelity than the reference build. We replaced the scale search with per-group alternating least-squares fitting (codes → LS-refit scale/zero → re-encode, iterated), which reproduces reference-level fidelity (rel ≈ 0.09–0.10 per module, verified against base fp16 weights).

Measured results

Quality gates: every module ≤ 0.11 relative error vs base weights (stock-dequant reference); 5-prompt A/B against the reference checkpoint (greedy, code / history / math / translation) — equivalent coherence, byte-identical on most prompts; IFEval (lm-eval local-completions, n=300, seed 42) 29.0 / 26.3 prompt-level loose/strict, 44.3 / 42.5 instruction-level.

The throughput figures above require the companion gfx1030 kernel patches (ikantkode/gfx1030-vllm-0.26 on GitHub). On other hardware this checkpoint still benefits any vLLM AWQ path via the reduced weight traffic.

Usage

vllm serve ikantkode/Qwen3.5-4B-AWQ-vd --dtype float16 --max-model-len 8192

Or one-command on a Radeon PRO V620 / gfx1030: ikantkode/Qwen3.5-vLLM-Deploy (Docker, ~98 tok/s out of the box).

Files & lineage

• Base: Qwen/Qwen3.5-4B (Apache-2.0)
• Quantization: AWQ INT4, group_size 128, asymmetric zero-point, GEMM packing; per-group scales/zeros fitted by alternating least-squares; no LN smoothing
• model_mtp.safetensors: MTP head for speculative decoding (qwen3_next_mtp). Note: on gfx1030 with heavily M=1-optimized decode kernels, MTP verification was measurably slower than plain decoding; on other stacks it may help.
• Full technical trail: ikantkode/gfx1030-vllm-0.26 (README + PROGRESS log)

Public benchmarks (lm-evaluation-harness)

Measured post-quantization, on the served checkpoint as-is — no fine-tuning, calibration recovery, or any kind of performance restoration was performed.

• Harness: EleutherAI lm-evaluation-harness
• Backend: live vLLM server (deployed configuration), greedy decoding, n=300/task, seed 42
• Protocol note: all scores use the raw completions endpoint (no chat template). Chat-mode vendor numbers (e.g. Qwen's IFEval 89.8) are not directly comparable — expect a large harness/protocol gap unrelated to quantization.

All multiple-choice/reasoning tasks sit in the expected band for a 4B-class model; no quantization-collapse signature. Development-time gates: per-module ≤ 0.11 rel-err vs base weights; greedy A/B vs the reference checkpoint.

Reproduce / related

• Quantization recipe (how this checkpoint was built): ikantkode/awq-quant-recipe
• Serving kernels + one-command deploy (v1.0.0 = rungs 16+18): ikantkode/gfx1030-vllm-0.26
• Prebuilt images: ghcr.io/ikantkode/gfx1030-vllm-0.26:v1.0.0 and docker.io/exaroot/gfx1030-vllm-0.26

Revisions

• v1.1.0 (Aug 19, 2026): untied lm_head quantized to INT4 (embeddings remain fp16; tie_word_embeddings now false). All 6 gates PASS including IFEval regression (29.0 → 29.3, sub-noise) and perplexity (+1.45%). Single-stream 84.5 → 97.9 tok/s (+15.9%); 16-user aggregate +27.6%; 128-user peak unchanged (the head cost amortizes across the batch at high concurrency). Docker image unchanged from v1.0.0 — only the checkpoint changed.
• v1.0.0 (Aug 18, 2026): rungs 16+18 multi-user knee (673 → 966.6 tok/s at 128 users); original full-INT4 release.