Downloads · 30 days
831
0% of all-time downloads
Mistral-7B-Instruct-v0.3-GPTQ-4bit
RedHatAI/Mistral-7B-Instruct-v0.3-GPTQ-4bit
Mistral-7B-Instruct-v0.3-GPTQ-4bit is a text generation model from RedHatAI. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
- Weight-only quantization of Mistral-7B-Instruct-v0.3 via GPTQ to 4bits with groupsize=128 - GPTQ optimized for 99.75% accuracy recovery relative to the unquantized model
All-time downloads
1.4M
Public
Parameters
7.2B
4.2 GB on disk
Likes
25
Public
Repo makeup
Click a slice to open those files.
.safetensors4.2 GB · 100%
Parameter types
How the weights are stored.
I327B · 96%
Try a prompt
Base models
Model card
From the Hugging Face model README
Model Card for Mistral-7B-Instruct-v0.3 quantized to 4bit weights
- Weight-only quantization of Mistral-7B-Instruct-v0.3 via GPTQ to 4bits with group_size=128
- GPTQ optimized for 99.75% accuracy recovery relative to the unquantized model
Open LLM Leaderboard evaluation scores
| Mistral-7B-Instruct-v0.3 | Mistral-7B-Instruct-v0.3-GPTQ-4bit<br>(this model) | |
|---|---|---|
| arc-c<br>25-shot | 63.48 | 63.40 |
| mmlu<br>5-shot | 61.13 | 60.89 |
| hellaswag<br>10-shot | 84.49 | 84.04 |
| winogrande<br>5-shot | 79.16 | 79.08 |
| gsm8k<br>5-shot | 43.37 | 45.41 |
| truthfulqa<br>0-shot | 59.65 | 57.48 |
| Average<br>Accuracy | 65.21 | 65.05 |
| Recovery | 100% | 99.75% |
vLLM Inference Performance
This model is ready for optimized inference using the Marlin mixed-precision kernels in vLLM: https://github.com/vllm-project/vllm
Simply start this model as an inference server with:
python -m vllm.entrypoints.openai.api_server --model neuralmagic/Mistral-7B-Instruct-v0.3-GPTQ-4bit
- Task
- Text Generation
- Library
- transformers
- Type
- mistral
- License
- apache-2.0
- Created
- May 23, 2024
- Updated
- Jun 10, 2024