Downloads · 30 days
11
2% of all-time downloads
jeffcookio/Mistral-Small-3.1-24B-Instruct-2503-HF-gptqmodel-4b-128g
Mistral-Small-3.1-24B-Instruct-2503-HF-gptqmodel-4b-128g is a machine learning model from jeffcookio. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
This is a Mistral-Small-3.1-24B-Instruct-2503 quantized from a hacked-up GPTQModel that has preliminary Mistral3ForConditionalGeneration support. There were several weird changes. Calibration was run against the flick…
Downloads · 30 days
11
2% of all-time downloads
All-time downloads
505
Public
Parameters
24B
15.2 GB on disk
Likes
2
Public
Click a slice to open those files.
.safetensors15.1 GB · 100%
How the weights are stored.
I3222.2B · 93%
From the Hugging Face model README
This is a Mistral-Small-3.1-24B-Instruct-2503 quantized from a hacked-up GPTQModel that has preliminary Mistral3ForConditionalGeneration support. There were several weird changes. Calibration was run against the flickr30k dataset (with too few samples; may upload a version with more significant calibration soon), and thus this should be a true vision-aware quant of the Mistral Small 3.1 HF checkpoint.
You need this branch of vLLM to run: https://github.com/sjuxax/vllm/tree/Mistral3.1
Another "feature" of this version is that it was quantized with a preliminary implementation of block-diagonal Hessians (which was authored entirely by Grok3). This allowed me to compute the quantization without OOM in my 24G VRAM.