mistral-small-3.2-24B-Instruct-2506-GGUF

Mistral-Small-3.2-24B-Instruct-2506 — GGUF Quantizations

This repository contains GGUF quantizations of Mistral-Small-3.2-24B-Instruct-2506, a 24B parameter multimodal model with Pixtral-style vision capabilities. These quantizations are optimized for AMD RDNA 3 (gfx1100) GPUs (RX 7900 XTX, RX 7900 XT, RX 7900 GRE) using llama.cpp with ROCm/Vulkan backends.

Three K-quant variants are provided — Q4_L, Q4_M, and Q4_S — offering a quality–size tradeoff. All were quantized from the BF16 GGUF baseline on a NVIDIA RTX PRO 6000 (Blackwell) and benchmarked on 2× AMD RX 7900 XTX.

Model Details

Quantization Variants

Quality Benchmarks

All benchmarks use wikitext-2-raw-v1 (test split) — the standard dataset for quantization quality evaluation, matching the methodology of llama.cpp, AutoAWQ, GPTQ, and the academic literature.

WikiText-2 Perplexity (ctx=512)

Measured via llama-perplexity b8984, 588 chunks.

* Q4_M PPL < F16 PPL is a known artifact of quantized models on wikitext-2 (token distribution shift). KLD is the reliable quality metric.

KL Divergence vs F16 (Static / Prefill)

KL divergence measures how much the output probability distribution has shifted from the F16 baseline. Lower is better; 0 = identical to F16.

Methodology: wikitext-2-raw-v1, ctx=512, full-vocab KLD computed on the second half of each 512-token chunk (positions [256–511]), ensuring every scored token has ≥256 tokens of left context. KLD direction: KL(P_base ‖ P_quant) — "how well does the quantized model approximate the base?"

Token Probability Divergence (Δp)

Same top-p = percentage of tokens where quantized and F16 models agree on the most likely next token.

Quality Ranking

Throughput Benchmarks (2× AMD RX 7900 XTX, gfx1100)

Benchmarks run on (2× RX 7900 XTX, ROCm 6.4.4) using llama.cpp llama-server with flash attention enabled.

Launch Configuration

llama-server -m <model>.gguf -c 131072 -ngl 99 -fa on \
  --host 0.0.0.0 --cache-type-k f16 --cache-type-v f16 \
  --tensor-split 1,1 --no-mmap -t 23 -ub 256 -b 256 --parallel <N>

Single-Request Throughput (parallel=1)

Multi-Request Throughput (parallel=8)

Detailed Per-Model Results

Q4_S — Fastest

Single request (parallel=1, ctx=131072):

8 concurrent (parallel=8, ctx=196608):

Q4_M — Balanced

Single request (parallel=1, ctx=131072):

8 concurrent (parallel=8, ctx=196608):

Q4_L — Best Quality

Single request (parallel=1, ctx=131072):

8 concurrent (parallel=8, ctx=180000):

Hardware Requirements

AMD GPUs (ROCm / Vulkan)

For dual-GPU setups (2× RX 7900 XTX), use --tensor-split 1,1 and --ngl 99 to distribute layers across both GPUs.

NVIDIA GPUs (CUDA)

These GGUF models also work on NVIDIA GPUs via llama.cpp CUDA backend. For NVIDIA deployment, consider the AutoRound W4A16 or NVFP4A16 quantizations instead — they offer better throughput on CUDA via vLLM.

Usage with llama.cpp

Server Mode (Recommended)

# AMD (ROCm):
./llama-server -m mistral-small-3.2-24b-instruct-Q4_M_AMD.gguf \
  -c 131072 -ngl 99 -fa on \
  --port 8000 --host 0.0.0.0 \
  --cache-type-k f16 --cache-type-v f16 \
  --tensor-split 1,1 --no-mmap -t 23 \
  -ub 256 -b 256 --parallel 4 \
  --mmproj mmproj-Q8_0.gguf

# NVIDIA (CUDA):
./llama-server -m mistral-small-3.2-24b-instruct-Q4_M_AMD.gguf \
  -c 131072 -ngl 99 -fa on \
  --port 8000 --host 0.0.0.0 \
  -ub 256 -b 256 --parallel 4 \
  --mmproj mmproj-Q8_0.gguf

Inference Test

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"mistral-small-3.2-24b-instruct-Q4_M_AMD","messages":[{"role":"user","content":"What is 2+2? One word."}],"max_tokens":10}'

Vision Test

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model":"mistral-small-3.2-24b-instruct-Q4_M_AMD",
    "messages":[
      {"role":"user","content":[{"type":"image_url","image_url":{"url":"https://upload.wikimedia.org/wikipedia/commons/thumb/4/47/PNG_transparency_demonstration_1.png/280px-PNG_transparency_demonstration_1.png"}},{"type":"text","text":"Describe this image in one sentence."}]}
    ],
    "max_tokens":100
  }'

Notes

Vision: Image Size Limit

The original Mistral model has max_image_size set to 1540. Images with dimensions exceeding the limit are proportionally downscaled before vision encoding. The mmproj files in this repository match the original specification.

Files in This Repository

License

This quantization is released under the Apache 2.0 License, following the base model's license.

The base model mistralai/Mistral-Small-3.2-24B-Instruct-2506 is licensed under Apache 2.0.

Citation

If you use this model in your research, please cite:

@misc{mistral-small-3.2-24b-gguf-gfx1100,
  title = {Mistral-Small-3.2-24B-Instruct-2506 GGUF Quantizations},
  author = {Gratex International},
  year = {2026},
  howpublished = {\url{https://huggingface.co/gratex/Mistral-Small-3.2-24B-Instruct-2506-GGUF}},
  note = {Quantized with llama.cpp, benchmarked on AMD RX 7900 XTX}
}

Acknowledgments

This quantization was produced using hardware provided by Gratex International, a.s.

---

Original Model: mistralai/Mistral-Small-3.2-24B-Instruct-2506 Quantization Tool: llama.cpp Inference Engine: llama.cpp (ROCm / CUDA / Vulkan)