Qwen3.5-4B-W4A16-AutoRound-LLM-Compressor

J-Fraudster/Qwen3.5-4B-W4A16-AutoRound-LLM-Compressor

This is a W4A16 (4-bit weight, 16-bit activation) quantized version of Qwen/Qwen3.5-4B, produced using AutoRound — Intel's sign gradient descent based quantization method designed for production-grade accuracy retention. MTP Enabled model quantization

Quantization Details

Key Notes

• High accuracy configuration — 1000 iterations with 512 calibration samples targets production-grade quality with minimal degradation from the base model.
• W4A16 — Weights are quantized to 4-bit integers; activations remain in FP16 for inference stability.
• ~50% memory reduction compared to the FP16 base model, enabling deployment on consumer and mid-range GPUs.
• Vision Tower (quant_nontext_module): False (Kept in BF16 to preserve visual reasoning and OCR precision)
• Special Modules (layer_config): Multi-Token Prediction (mtp, mtp.fc) kept in native bfloat16

MTP / Speculative Decoding

This model supports Multi-Token Prediction (MTP) for improved inference throughput using speculative decoding.

When serving with compatible backends (e.g., vLLM), enable MTP using:

--speculative_config '{"method":"mtp","num_speculative_tokens":1}'

Notes

• num_speculative_tokens=1 is a stable default for balancing speed and accuracy.
• You can experiment with higher values for better throughput, depending on your hardware and latency requirements.

Usage

This model is compatible with transformers and backends that support AutoRound GPTQ-format weights (e.g., vLLM, SGLang, AutoGPTQ). For full model details, architecture, and capabilities, refer to the base model page.

---