Qwen3.5-0.8B-W4A16-AutoRound-LLM-Compressor

Vishva007/Qwen3.5-0.8B-W4A16-AutoRound-LLM-Compressor

This is a W4A16 (4-bit weight, 16-bit activation) GPTQ-format quantized version of Qwen/Qwen3.5-0.8B, produced using AutoRound — Intel's sign gradient descent based quantization method designed for production-grade accuracy retention.

Quantization Details

Key Notes

• GPTQ format — Exported in the standard GPTQ format for broad ecosystem compatibility.
• Ultra-high accuracy configuration — 1000 iterations with 512 calibration samples ensures near-lossless quantization, especially critical at this model scale where parameter budget is tight.
• W4A16 — Weights are quantized to 4-bit integers; activations remain in FP16 for inference stability.
• Extremely lightweight — The quantized 0.8B model is suitable for edge deployment, low-latency inference, and resource-constrained environments.
• ~50% memory reduction compared to the FP16 base model.

Usage

This model is compatible with transformers, AutoGPTQ, vLLM, and SGLang — any backend supporting GPTQ-format weights works out of the box. For full model details, architecture, and capabilities, refer to the base model page.

🚀 Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.

🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.

PyTorch 2.14

PyTorch 2.13

PyTorch 2.12

---