Downloads · 30 days
5
22% of all-time downloads
quantized-mistral-7b
wjleece/quantized-mistral-7b
quantized-mistral-7b is a machine learning model from wjleece. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
This is a 4-bit quantized version of Mistral-7B-Instruct-v0.1.
All-time downloads
23
Public
Parameters
7.4B
4.5 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors4.5 GB · 100%
Parameter types
How the weights are stored.
U87B · 95%
Model card
From the Hugging Face model README
This is a 4-bit quantized version of Mistral-7B-Instruct-v0.1.
I loaded this model to HuggingFace so that I didn't have to spend Colab GPUs to continually (re)quantize the same model.
The quantized LLM was created via:
from huggingface_hub import upload_folder
#Load the pre-quantized model and tokenizer quantization_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16 )
#Load the quantized model from Colab environment model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1", quantization_config=quantization_config, device_map='auto')
#Save the quantized model to a local directory model.save_pretrained("wjleece-quantized-mistral-7b")
#Upload the quantized model to Hugging Face
upload_folder(
folder_path="wjleece-quantized-mistral-7b",
repo_id="wjleece/quantized-mistral-7b",
use_auth_token=True
)
- Type
- mistral
- License
- apache-2.0
- Created
- Oct 24, 2024
- Updated
- Oct 25, 2024