mistral-7b-4bit

Mistral-7B-Instruct-v0.3 (4-bit Quantized)

🚀 Overview

This repository provides a 4-bit quantized version of Mistral-7B-Instruct-v0.3, optimized for faster inference and lower memory usage while maintaining high-quality instruction-following capabilities.

Quantization is achieved using BitsAndBytes (bnb) with 4-bit precision, making it feasible to run this model on consumer GPUs (such as T4 or A10G) and CPU inference environments.

---

🛠 Model Details

• Base Model: mistralai/Mistral-7B-Instruct-v0.3
• Quantization Type: 4-bit (bnb)
• Library: transformers
• Hardware Requirements:
  • ✅ GPU: Runs on NVIDIA T4 (16GB VRAM), A100 (40GB VRAM), or higher.
  • ✅ CPU: Can run inference on CPU (slower than GPU).
• Use Cases: Chatbots, Retrieval-Augmented Generation (RAG), Instruction-following NLP tasks.

---

🔧 Installation

Ensure you have the necessary dependencies:

pip install transformers bitsandbytes accelerate

---

📥 Usage

1️⃣ Load the Model in 4-bit Quantization

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

MODEL_NAME = "Noorhan/mistral-7b-4bit"

# Configure quantization
quantization_config = BitsAndBytesConfig(load_in_4bit=True)

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME, 
    device_map="auto", 
    quantization_config=quantization_config
)

# Generate response
prompt = "Explain quantization in machine learning."
inputs = tokenizer(prompt, return_tensors="pt").to("cuda" if torch.cuda.is_available() else "cpu")
outputs = model.generate(**inputs, max_new_tokens=150)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

🔥 Why Use This Quantized Model?

✅ Lower Memory Usage - Reduces VRAM needs from ~35GB to ~5GB
✅ Faster Inference - Enables deployment on cloud GPUs (T4, A10G)
✅ Minimal Quality Loss - Maintains strong performance in instruction-following tasks
✅ Hugging Face Integration - Can be easily loaded with transformers

---

⚡ Benchmarking & Performance

---

📝 Use Cases

• 🚀 Retrieval-Augmented Generation (RAG)
• 💬 Chatbots & Conversational AI
• 🧠 Prompt Engineering & In-Context Learning
• 🎓 Educational NLP Applications
• 🤖 Low-Cost AI Model Deployment

---

📌 Limitations

• 🔸 Lower Precision: 4-bit quantization slightly reduces accuracy compared to FP16.
• 🔸 Longer Sequences: Performance may degrade for very large text generations.
• 🔸 Limited Compatibility: Not all transformer optimizations work with bnb quantization.

---

📜 License

This model follows the licensing of mistralai/Mistral-7B-Instruct-v0.3. Check Mistral AI’s terms of use.

---

🔗 Citations

If you use this model, please cite:

@misc{mistral7b-4bit,
  title={Mistral-7B-Instruct-v0.3 (4-bit Quantized)},
  author={Noorhan},
  year={2024},
  publisher={Hugging Face},
  url={https://huggingface.co/Noorhan/mistral-7b-4bit}
}