Downloads · 30 days
4
9% of all-time downloads
V-Bharath-raj/qwen2.5-0.5b-int8-quanto
qwen2.5-0.5b-int8-quanto is a machine learning model from V-Bharath-raj. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
This repository contains an INT8 quantized version of the Qwen2.5-0.5B-Instruct model using Optimum Quanto. The model is optimized for reduced memory usage and improved inference efficiency while maintaining good gene…
Downloads · 30 days
4
9% of all-time downloads
All-time downloads
44
Public
Parameters
631M
779 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors767 MB · 99%
How the weights are stored.
I8494M · 78%
From the Hugging Face model README
This repository contains an INT8 quantized version of the Qwen2.5-0.5B-Instruct model using Optimum Quanto. The model is optimized for reduced memory usage and improved inference efficiency while maintaining good generation quality.
This model was quantized using Optimum Quanto with INT8 precision.
| Type | Precision |
|---|---|
| Original Model | FP32 |
| Quantized Model | INT8 |
| Version | Size |
|---|---|
| Original Model | ~1.39 GB |
| Quantized Model | ~767 MB |
The quantization process was performed on:
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "V-Bharath-raj/qwen2.5-0.5b-int8-quanto"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
prompt = "Explain quantization in simple terms."
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
**inputs,
max_new_tokens=100
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
This repository inherits the license and usage restrictions of the original Qwen2.5-0.5B-Instruct model.