Downloads · 30 days
24
2% of all-time downloads
Mistral-NeMo-Instruct-2407-GGUF
NLPoetic/Mistral-NeMo-Instruct-2407-GGUF
Mistral-NeMo-Instruct-2407-GGUF is a machine learning model from NLPoetic. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
This repository contains four quantized versions of Mistral-NeMo-Instruct-2407, created using llama.cpp. The goal was to examine how different quantization methods affect prompt sensitivity with sentiment classificati…
All-time downloads
1.2K
Public
Repo size
43.5 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.gguf43.5 GB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Oct 17, 2024
- Updated
- Oct 25, 2024
- SHA
- b50b1772
Base models
Model card
From the Hugging Face model README
Quantized Mistral-NeMo-Instruct-2407 versions for Prompt Sensitivity Blog
This repository contains four quantized versions of Mistral-NeMo-Instruct-2407, created using llama.cpp. The goal was to examine how different quantization methods affect prompt sensitivity with sentiment classification tasks.
Quantization Details
Models were quantized using llama.cpp (release b3922). The imatrix versions used an imatrix.dat file created from Bartowski's calibration dataset, mentioned here.
Models
| Filename | Size | Description |
|---|---|---|
| Mistral-NeMo-12B-Instruct-2407-Q8_0.gguf | 13 GB | 8-bit default quantization |
| Mistral-NeMo-12B-Instruct-2407-Q5_0.gguf | 8.73 GB | 5-bit default quantization |
| Mistral-NeMo-12B-Instruct-2407-imatrix-Q8_0.gguf | 13 GB | 8-bit with imatrix quantization |
| Mistral-NeMo-12B-Instruct-2407-imatrix-Q5_0.gguf | 8.73 GB | 5-bit with imatrix quantization |
I've also included the imatrix.dat (7.05 MB) file used to create the imatrix-quantized versions.
Findings
Prompt sensitivity was seen specifically in 5-bit models using imatrix quantization, but not with default llama.cpp quantization settings. Prompt sensitivity was not observed in 8-bit models with either quantization method.
For further discussion please see my accompanying blog post.
Author
Simon Barnes
- License
- apache-2.0
- Languages
- en
- Created
- Oct 17, 2024
- Updated
- Oct 25, 2024