mistral-24b-4bit

Mistral-Small-24B-Instruct-2501 (4-bit Quantized)

🚀 Overview

This repository provides a 4-bit quantized version of Mistral-Small-24B-Instruct-2501, optimized for faster inference and lower memory usage while maintaining strong performance in instruction-following tasks.

Quantization is performed using BitsAndBytes (bnb) 4-bit precision, making it feasible to run this large-scale model on consumer GPUs (such as A100 40GB, A10G, or even T4 in limited use cases).

---

🛠 Model Details

• Base Model: mistralai/Mistral-Small-24B-Instruct-2501
• Quantization Type: 4-bit (bnb)
• Library: transformers
• Hardware Requirements:
  • ✅ GPU: Runs on NVIDIA A100 (40GB VRAM), A10G (24GB VRAM) or higher.
  • ✅ Limited CPU Use: Possible on CPU for small inference tasks, but much slower.
• Use Cases: Advanced NLP tasks, Retrieval-Augmented Generation (RAG), chatbots, in-context learning, and fine-tuning for domain-specific applications.

---

🔧 Installation

Ensure you have the necessary dependencies:

pip install transformers bitsandbytes accelerate

---

📥 Usage

1️⃣ Load the Model in 4-bit Quantization

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

MODEL_NAME = "Noorhan/mistral-24b-4bit"

# Configure quantization
quantization_config = BitsAndBytesConfig(load_in_4bit=True)

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME, 
    device_map="auto", 
    quantization_config=quantization_config
)

# Generate response
prompt = "Explain the concept of transformers in deep learning."
inputs = tokenizer(prompt, return_tensors="pt").to("cuda" if torch.cuda.is_available() else "cpu")
outputs = model.generate(**inputs, max_new_tokens=150)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

🔥 Why Use This Quantized Model?

✅ Lower Memory Usage - Reduces VRAM needs from ~50GB to ~10GB
✅ Faster Inference - Optimized for cloud GPUs (A100, A10G, and T4)
✅ Minimal Quality Loss - Maintains strong performance in instruction-following tasks
✅ Hugging Face Integration - Can be easily loaded with transformers

---

⚡ Benchmarking & Performance

---

📝 Use Cases

• 🔍 Retrieval-Augmented Generation (RAG)
• 💬 Chatbots & Conversational AI
• 🧠 Prompt Engineering & In-Context Learning
• 🎓 Educational NLP Applications
• 🏥 Medical and Legal Document Processing
• 🤖 Enterprise AI Assistants

---

📌 Limitations

• 🔸 Higher VRAM Requirements: Despite being quantized, this model still requires a high-end GPU (e.g., A100, A10G).
• 🔸 Lower Precision: 4-bit quantization slightly reduces accuracy compared to FP16.
• 🔸 Longer Sequences: Performance may degrade for very large text generations.
• 🔸 Limited CPU Inference: CPU inference is possible but significantly slower.

---

📜 License

This model follows the licensing of mistralai/Mistral-Small-24B-Instruct-2501. Check Mistral AI’s terms of use.

---

🔗 Citations

If you use this model, please cite:

@misc{mistral24b-4bit,
  title={Mistral-Small-24B-Instruct-2501 (4-bit Quantized)},
  author={Noorhan},
  year={2024},
  publisher={Hugging Face},
  url={https://huggingface.co/Noorhan/mistral-small-24b-4bit}
}