Downloads · 30 days
9
19% of all-time downloads
quantized-mistral-nemo-12b
wjleece/quantized-mistral-nemo-12b
quantized-mistral-nemo-12b is a machine learning model from wjleece. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
This is a 4-bit quantized version of mistralai/Mistral-Nemo-Instruct-2407.
All-time downloads
47
Public
Parameters
12.4B
8.8 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors8.8 GB · 100%
Parameter types
How the weights are stored.
U810.9B · 88%
Model card
From the Hugging Face model README
This is a 4-bit quantized version of mistralai/Mistral-Nemo-Instruct-2407.
I loaded this model to HuggingFace so that I didn't have to spend Colab GPUs to continually (re)quantize the same model.
The quantized LLM was created via:
from huggingface_hub import upload_folder
#Load the pre-quantized model and tokenizer quantization_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16 )
#Load the quantized model from Colab environment model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-Nemo-Instruct-2407", quantization_config=quantization_config, device_map='auto')
#Save the quantized model to a local directory model.save_pretrained("wjleece-quantized-mistral-nemo-12b")
#Upload the quantized model to Hugging Face
upload_folder(
folder_path="wjleece-quantized-mistral-nemo-12b",
repo_id="wjleece/quantized-mistral-nemo-12b",
use_auth_token=True
)
- Type
- mistral
- License
- apache-2.0
- Created
- Oct 25, 2024
- Updated
- Oct 25, 2024