Qwen_Quantised3.50.5b

---

language:

• en license: apache-2.0 library_name: transformers tags:
• quantization
• optimum-quanto
• qwen
• int8
• cpu pipeline_tag: text-generation base_model: Qwen/Qwen2.5-0.5B-Instruct

---

Model Quantisation (Qwen2.5-0.5B, CPU Only)

This technique is based on Post-Training Quantization (PTQ) using Qwen2.5-0.5B-Instruct...

---

language:

• en license: apache-2.0 library_name: transformers pipeline_tag: text-generation base_model: Qwen/Qwen2.5-0.5B-Instruct tags:
• quantization
• int8
• cpu
• optimum-quanto
• transformers model_type: qwen2

---

Model Quantisation (Qwen2.5-0.5B, CPU Only)

This Technique based on the PQT Post Quantization Training A model quantisation using Qwen2.5-0.5B-Instruct. The project shows how to quantise a model with Optimum Quanto and run it locally on a CPU. where a 32 bit Modal into 8 bit

Files

Setup

Install dependencies:

pip install -r requirements.txt

Usage

1. Quantise the model

HuggingFace Token Need
$env:HF_TOKEN = "Your TOKEN"; python quant_qwen.py 
python quant_qwen.py 

This creates:

qwen-int8/

2. Run the quantised model

 python .\v1_quant_qwen.py      

Please enter your question: hi

INPUT : hi OUTPUT: Hello! How can I assist you today? Please let me know if there's anything specific you'd like to talk about or any questions you have. I'm here to help answer your queries.

3. Open the notebook

jupyter notebook 01_model_quantisation_guide.ipynb

What is Quantisation?

Quantisation reduces the precision of model weights (for example, FP32 → INT8).

Benefits:

• Smaller model size
• Lower memory usage
• Faster inference
• CPU-friendly deployment

Requirements

• Python 3.10+
• CPU-only machine supported
• No NVIDIA GPU required