Qwen3.5-4B-AWQ

Qwen3.5-4B-AWQ (W4A16)

AWQ 4-bit (W4A16) quantization of Qwen/Qwen3.5-4B, built with llm-compressor in compressed-tensors format for native vLLM loading (Marlin kernels).

Qwen3.5-4B is a hybrid-attention VLM: interleaved full self-attention + Gated-DeltaNet linear attention, a vision tower, and an MTP head for speculative decoding.

What is quantized

• 248 Linear layers quantized; on-disk ~5.0 GB (vs ~8 GB BF16).

Quality (AWQ vs BF16)

OpenLLM-lite, n=200 per task, greedy logprob MC scoring on RTX 5060 Ti (sm_120):

Mean MC recovery: 98.1%.

Evals

Full AWQ vs BF16 dump: LostGentoo/awq-quant-evals (qwen35_4b_awq_vs_bf16.json, plus combined quant_quality_evals.json).

Optimal vLLM serve

compressed-tensors is auto-detected - do not pass --quantization awq.

Recommended (text + vision, MTP on)

vllm serve LostGentoo/Qwen3.5-4B-AWQ \
  --trust-remote-code \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --limit-mm-per-prompt '{"image":1}' \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --generation-config vllm \
  --mamba-cache-mode align \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Notes:

• MTP: requires model_mtp.safetensors (included). Older docs may say qwen3_5_mtp; current vLLM remaps that to mtp.
• --mamba-cache-mode align: needed for hybrid Gated-DeltaNet + MTP / prefix-cache paths (all is unsupported for Qwen3.5 MTP).
• Thinking: default Qwen3.5 thinking can burn tokens; keep it off unless you want CoT. For hard math/coding, set "enable_thinking": true and raise max_tokens.
• Vision: keep --limit-mm-per-prompt '{"image":1}' so the VLM path is enabled; omit only for text-only deployments.
• Blackwell (sm_120): Marlin W4A16 works; add --enforce-eager only if first bring-up hits compile issues.

Python

from vllm import LLM, SamplingParams

llm = LLM(
    model="LostGentoo/Qwen3.5-4B-AWQ",
    trust_remote_code=True,
    max_model_len=8192,
    limit_mm_per_prompt={"image": 1},
    speculative_config={"method": "mtp", "num_speculative_tokens": 3},
)
sp = SamplingParams(temperature=0.7, top_p=0.8, top_k=20, max_tokens=256)
print(llm.generate(["Explain entropy in one sentence."], sp)[0].outputs[0].text)

OpenAI-compatible client

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "LostGentoo/Qwen3.5-4B-AWQ",
    "messages": [{"role":"user","content":"In one sentence, what is entropy?"}],
    "max_tokens": 128,
    "temperature": 0.7,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

Recipe

• llm-compressor 0.12: AWQModifier(duo_scaling=False) + QuantizationModifier(W4A16)
• Ignore: re:.*visual.*, re:.*lm_head, re:.*mtp.*
• Calib: 256 mixed-modal samples from lmms-lab/flickr30k, seq 2048
• MTP splice after save from Qwen/Qwen3.5-4B

License

Apache-2.0, inherited from the base model.