Downloads · 30 days
7
5% of all-time downloads
Mistral-7B-v0.3-GGUF
Arivukkarasu/Mistral-7B-v0.3-GGUF
Mistral-7B-v0.3-GGUF is a machine learning model from Arivukkarasu. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for llama.cpp. The card lists the license as mit.
This is a quantized GGUF version of mistralai/Mistral-7B-v0.3, quantized to Q4KM using llama.cpp's quantization tool.
All-time downloads
153
Public
Repo size
4.4 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.gguf4.4 GB · 100%
At a glance
- Library
- llama.cpp
- License
- mit
- Access
- Public
- Created
- Aug 26, 2025
- Updated
- Aug 26, 2025
- SHA
- cc53011c
Base models
Model card
From the Hugging Face model README
Mistral-7B - Q4_K_M GGUF
This is a quantized GGUF version of mistralai/Mistral-7B-v0.3, quantized to Q4_K_M using llama.cpp's quantization tool.
Model Info
- Base model: Mistral-7B-v0.3
- Quantization: Q4_K_M
- Format: GGUF (compatible with llama.cpp and llama-cpp-python)
- Size: ~4.2 GB (approximate, may vary)
- Use case: General-purpose language generation, chatbots, assistants, and instruction-following tasks optimized for CPU inference
📦 Files
Mistral-7B-v0.3-Q4_K_M.gguf: The quantized model
How to Use (with llama-cpp-python)
from llama_cpp import Llama
llm = Llama(model_path="Mistral-7B-Q4_K_M.gguf")
output = llm("Explain quantum computing in simple terms.", max_tokens=200)
print(output)
Recommended Settings
- Context size: 4096 tokens (depending on your llama.cpp version)
- Hardware: Optimized for CPU (AVX2 or better recommended); also runs efficiently on GPU with llama.cpp compiled with CUDA or Metal support
Credits
- Original model: mistralai/Mistral-7B-Instruct-v0.3
- Quantized with: llama.cpp
- Library
- llama.cpp
- License
- mit
- Created
- Aug 26, 2025
- Updated
- Aug 26, 2025