Qwen2.5-7B-Instruct-INT8-Quanto

This model has been pushed to the Hub using the PytorchModelHubMixin integration.

• Base Model: Qwen/Qwen2.5-7B-Instruct
• Quantization: INT8 (Optimum Quanto)
• Library: Optimum Quanto
• Framework: PyTorch
• Documentation: https://huggingface.co/docs/optimum/quanto

---

Qwen2.5-7B-Instruct INT8 (Optimum Quanto)

This repository contains an INT8 quantized version of Qwen2.5-7B-Instruct created using Optimum Quanto.

The objective of this project is to reduce model size and memory usage while maintaining inference quality for research and local deployment.

Model Details

• Base Model: Qwen/Qwen2.5-7B-Instruct
• Quantization Method: Optimum Quanto
• Weight Precision: INT8
• Original Model Size: ~15.2 GB
• Quantized Model Size: ~8.18 GB
• Storage Reduction: ~46%

Usage

from optimum.quanto import QuantizedModelForCausalLM
from transformers import AutoTokenizer

model = QuantizedModelForCausalLM.from_pretrained(
    "YOUR_USERNAME/Qwen2.5-7B-Instruct-INT8"
)

tokenizer = AutoTokenizer.from_pretrained(
    "YOUR_USERNAME/Qwen2.5-7B-Instruct-INT8"
)

Benchmark

Benchmarking is currently in progress.

Planned evaluations include:

• HellaSwag
• ARC Challenge
• Winogrande
• GSM8K

Benchmark results will be added after evaluation.

Hardware

Quantization performed on:

• Windows
• Python 3.11
• PyTorch
• NVIDIA GeForce RTX 3050 Laptop GPU (4 GB VRAM)

Limitations

• Quantization may slightly reduce model accuracy depending on the evaluation task.
• Performance should be validated for production workloads.
• GPU memory requirements depend on sequence length and runtime configuration.

License

This repository contains a quantized derivative of Qwen/Qwen2.5-7B-Instruct.

Please follow the original model's license and usage terms when using or redistributing this model.

---

This model has been pushed to the Hugging Face Hub using:

• Framework: Optimum Quanto

• Base Model: Qwen/Qwen2.5-7B-Instruct

• Quantization: INT8

• Library: huggingface_hub + Optimum Quanto

• Documentation: https://huggingface.co/docs/optimum/quanto

• Author

• **Monishwaran K

• GitHub: https://github.com/Monishwaran45

• Hugging Face: https://huggingface.co/Helllbos ⭐ If you find this model useful, please consider giving it a Like ❤️ on Hugging Face.