Downloads · 30 days
8
6% of all-time downloads
mistral-small-int8
Parveshiiii/mistral-small-int8
mistral-small-int8 is a text generation model from Parveshiiii. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
This is a quantized variant of the Mistral 7B (small) model using LLM.int8() quantization via bitsandbytes. It reduces memory footprint while maintaining high-generation quality—ideal for single-GPU inference, researc…
All-time downloads
145
Public
Parameters
7.2B
7.5 GB on disk
Likes
1
Public
Repo makeup
Click a slice to open those files.
.safetensors7.5 GB · 100%
Parameter types
How the weights are stored.
I87B · 96%
Try a prompt
Base models
Model card
From the Hugging Face model README
🧠 Model Overview
This is a quantized variant of the Mistral 7B (small) model using LLM.int8() quantization via bitsandbytes. It reduces memory footprint while maintaining high-generation quality—ideal for single-GPU inference, research benchmarks, and lightweight downstream applications.
🔧 Model Specs
- Total Parameters: ~7 Billion
- Precision: INT8 with FP32 CPU offload
- Quantization Threshold: 6.0
- Device Map: Auto (compatible with CUDA / CPU offloading)
- Tokenizer: Byte-level BPE
🚀 Usage Example
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
model_id = "ParveshRawal/mistral-small-int8"
tokenizer = AutoTokenizer.from_pretrained(model_id)
quant_config = BitsAndBytesConfig(
load_in_8bit=True,
llm_int8_threshold=6.0,
llm_int8_enable_fp32_cpu_offload=True
)
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="auto",
quantization_config=quant_config
)
inputs = tokenizer("Tell me something about IndiaAI.", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
- Task
- Text Generation
- Library
- transformers
- Type
- mistral
- License
- mit
- Created
- Jul 8, 2025
- Updated
- Jul 8, 2025