Qwen3-8B-INT8

Qwen3-8B-Quanto-INT8

INT8 quantized derivative of Qwen3-8B using Optimum Quanto for efficient inference with reduced memory usage.

Model Description

This repository provides an INT8 quantized version of Qwen3-8B created using Optimum Quanto.

The objective of this project is to reduce model size and memory requirements while maintaining strong inference performance for real-world applications.

Features

• INT8 Quantization
• Reduced Memory Usage
• Faster Inference on Supported Hardware
• Optimized for AI Assistants
• Code Generation
• Question Answering
• Multilingual Support
• Compatible with Hugging Face Transformers

---

Base Model

---

Installation

pip install torch transformers optimum-quanto

---

Usage

from transformers import AutoTokenizer
from optimum.quanto import QuantizedModelForCausalLM

model_id = "Helllbos/Qwen3-8B-Quanto-INT8"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = QuantizedModelForCausalLM.from_pretrained(model_id)

prompt = "Explain Quantization."

inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=200
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

Recommended Use Cases

• AI Chatbots
• Personal AI Assistants
• Code Generation
• Educational Applications
• Research
• MLOps Workflows
• Local LLM Deployment
• API-Based Inference

---

Hardware Recommendations

Minimum

• 16 GB RAM
• Modern Intel or AMD CPU

Recommended

• NVIDIA CUDA-enabled GPU
• 16 GB+ RAM

Performance depends on hardware, prompt length, inference backend, and runtime configuration.

---

Limitations

• This repository contains a quantized derivative of the original Qwen3-8B model.
• Output quality may differ slightly from the original FP16 model due to quantization.
• Performance depends on the inference backend and available hardware.

---

License

This repository distributes a quantized derivative of the original Qwen3-8B model.

Please follow the original Qwen3-8B license and usage terms when using or redistributing this model.

---

Acknowledgements

• Qwen Team
• Hugging Face
• Optimum Quanto
• PyTorch
• Transformers

---

Author

Monishwaran K

• GitHub: https://github.com/Monishwaran45
• Hugging Face: https://huggingface.co/Helllbos

⭐ If you find this model useful, please consider giving it a Like ❤️ on Hugging Face.