Qwen3-4B-Int8-Dynamic

Qwen3-4B-Int8-Dynamic

Model Overview

INT8 Dynamic Quantized version of Qwen3-4B, quantized by Baidu AICAPX Team, optimized for Kunlun XPU.

🔗 Original Model: Qwen/Qwen3-4B

Quantization Details

This model is quantized using SmoothQuant + GPTQ pipeline with INT8 Dynamic Quantization (W8A8) method.

Quantization Configuration

<table style="text-align: center;"> <thead> <tr> <th colspan="2" style="text-align: center;">Symmetric Quant</th> <th colspan="2" style="text-align: center;">Asymmetric Quant</th> <th colspan="2" style="text-align: center;">Model Type</th> <th colspan="2" style="text-align: center;">Granularity</th> </tr> <tr> <th>weight</th> <th>activation</th> <th>weight</th> <th>activation</th> <th>Dense</th> <th>MoE</th> <th>weight</th> <th>activation</th> </tr> </thead> <tbody> <tr> <td>✅</td> <td>✅</td> <td>❌</td> <td>❌</td> <td>✅</td> <td>❌</td> <td>per-channel</td> <td>per-token</td> </tr> </tbody> </table>

Detailed Configuration:

{
  "quant_method": "compressed-tensors",
  "format": "int-quantized",
  "weights": {
    "num_bits": 8,
    "symmetric": true,
    "strategy": "channel",
    "observer": "minmax"
  },
  "input_activations": {
    "num_bits": 8,
    "symmetric": true,
    "strategy": "token",
    "dynamic": true
  }
}

Requirements

[!IMPORTANT] This model requires vLLM >= 0.11.0 for inference. For Kunlun XPU, please use vllm-kunlun >= 0.11.0.

Software Dependencies

• transformers >= 4.51.0
• vllm >= 0.11.0 (for NVIDIA GPU)
• vllm-kunlun >= 0.11.0 (for Kunlun XPU)

With transformers<4.51.0, you will encounter the following error:

KeyError: 'qwen3'

Deployment with SGLang

python -m sglang.launch_server --model-path Qwen3-4B-Int8-Dynamic --reasoning-parser qwen3

Best Practices

• Thinking mode: Temperature=0.6, TopP=0.95, TopK=20, MinP=0. DO NOT use greedy decoding.
• Non-thinking mode: Temperature=0.7, TopP=0.8, TopK=20, MinP=0.
• Set presence_penalty between 0–2 to reduce endless repetitions.
• For long texts (>32K tokens), enable YaRN for up to 131K context.

For more details on thinking/non-thinking modes, refer to Qwen3 Documentation.

Accuracy

Credits

• Original Model: Qwen/Qwen3-4B by Qwen Team
• Quantization: Baidu AICAPX Team
• Optimized for: Kunlun XPU with vllm-kunlun >= 0.11.0