Downloads · 30 days
6K
2% of all-time downloads
Mistral-Small-3.2-24B-Instruct-2506-awq-sym
jeffcookio/Mistral-Small-3.2-24B-Instruct-2506-awq-sym
Mistral-Small-3.2-24B-Instruct-2506-awq-sym is a machine learning model from jeffcookio. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Created with llm-compressor's latest changes, quantized on a GH200, works well for me with vLLM's main branch on my RTX 3090Ti as of 2025-07-01.
All-time downloads
324K
Public
Parameters
24.2B
20 GB on disk
Likes
12
Public
Repo makeup
Click a slice to open those files.
.safetensors15 GB · 100%
Parameter types
How the weights are stored.
I3222.2B · 92%
Base models
Datasets
Model card
From the Hugging Face model README
Created with llm-compressor's latest changes, quantized on a GH200, works well for me with vLLM's main branch on my RTX 3090Ti as of 2025-07-01.
What about tool calling?
Per https://vllm-dev.slack.com/archives/C07QP347J4D/p1751401629797809?thread_ts=1751399869.254259&cid=C07QP347J4D, there is currently no way to get tool calling with Mistral-HF formatted models.
I've worked around this on a GitHub branch here: https://github.com/sjuxax/vllm/tree/Mistral3.1-rebase . It includes code to remap the weights from HF-Mistral to Mistral, allowing use of MistralTokenizer.
I've updated the config.json to be compatible with this approach, and I'm about to push the tekken.json tokenizer. With that, if you build that branch, you should be able
to run this checkpoint with MistralTokenizer and get tool calling.
Note: I spoke a little too soon on the above. We also needed https://github.com/vllm-project/vllm/pull/20503 to get tool calling to work properly. I've merged and pushed this to the Mistral3.1-rebase branch.
- Type
- mistral3
- Created
- Jul 1, 2025
- Updated
- Jul 4, 2025