Downloads · 30 days
25
1% of all-time downloads
Mistral-7B-Instruct-v0.3-GGUF
mukel/Mistral-7B-Instruct-v0.3-GGUF
Mistral-7B-Instruct-v0.3-GGUF is a machine learning model from mukel. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
In the wild, Q80 quantizations are fine, but Q40 quantizations are rarely pure e.g. the output.weights tensor is quantized with Q6K, instead of Q40. A pure Q40 quantization can be generated from a high precision (F32,…
All-time downloads
2.1K
Public
Repo size
11.8 GB
Likes
2
Public
Repo makeup
Click a slice to open those files.
.gguf11.8 GB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- May 22, 2024
- Updated
- Jun 1, 2024
- SHA
- d1196197
Model card
From the Hugging Face model README
Pure quantizations of Mistral-7B-Instruct-v0.3 for mistral.java.
In the wild, Q8_0 quantizations are fine, but Q4_0 quantizations are rarely pure e.g. the output.weights tensor is quantized with Q6_K, instead of Q4_0.
A pure Q4_0 quantization can be generated from a high precision (F32, F16, BFLOAT16) .gguf source with the quantize utility from llama.cpp as follows:
./quantize --pure ./Mistral-7B-Instruct-v0.3-F32.gguf ./Mistral-7B-Instruct-v0.3-Q4_0.gguf Q4_0
Original model: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3
**Note that this model does not support a System prompt.
The Mistral-7B-Instruct-v0.3 Large Language Model (LLM) is an instruct fine-tuned version of the Mistral-7B-v0.3.
Mistral-7B-v0.3 has the following changes compared to Mistral-7B-v0.2
- Extended vocabulary to 32768
- Supports v3 Tokenizer
- Supports function calling
- License
- apache-2.0
- Created
- May 22, 2024
- Updated
- Jun 1, 2024