Mistral-Small-3.2-24B-Instruct-2506-W4A16-AutoRound

Mistral-Small-3.2-24B-Instruct-2506 — AutoRound W4A16 Quantization

This is a W4A16 (4-bit weight-only) quantization of Mistral-Small-3.2-24B-Instruct-2506, a 24B parameter multimodal model with Pixtral-style vision capabilities, quantized using AutoRound v0.10.2 with SignRound optimization (1000 iterations).

AutoRound W4A16 stores weights as INT4 with BF16 activations. Weights are dequantized via GPTQ-style group quantization (group_size=128) before matrix multiply. This format is widely supported across GPU architectures (Ampere, Ada, Blackwell) and inference engines (vLLM, SGLang, TensorRT-LLM).

Model Details

Quantization Configuration

{
  "bits": 4,
  "data_type": "int",
  "group_size": 128,
  "sym": true,
  "batch_size": 4,
  "iters": 1000,
  "low_gpu_mem_usage": true,
  "nsamples": 512,
  "block_name_to_quantize": "model.language_model.layers",
  "quant_method": "auto-round",
  "packing_format": "auto_round:auto_gptq"
}

Key parameters:

• iters=1000: Maximum SignRound optimization steps per block (~5× slower than default 200, best accuracy)
• nsamples=512: 512 calibration samples (4× default of 128)
• sym=true: Symmetric quantization (no zero-point)
• group_size=128: Per-128-element scaling groups

Calibration Dataset

The 512 calibration samples were built from a domain-specific mix of text-only datasets (no images/video, to avoid torchvision import errors in the llm-compressor environment). Short samples were concatenated into chunks of ≥2,048 Tekken tokens each:

Quality Benchmarks

All benchmarks use wikitext-2-raw-v1 (test split) — the standard dataset for quantization quality evaluation, matching the methodology of llama.cpp, AutoAWQ, GPTQ, and the academic literature.

WikiText-2 Perplexity (ctx=512)

Measured via vLLM completions API with echo=True + logprobs. 642 non-overlapping chunks, ~301K tokens scored. Methodology matches llama.cpp ./perplexity -c 512.

KL Divergence vs BF16 (Static / Prefill)

KL divergence measures how much the output probability distribution has shifted from the base model. Lower is better; 0 = identical.

Methodology matches llama.cpp --kl-divergence: wikitext-2-raw-v1, ctx=512, score only the second half of each chunk (positions [256–511]), which ensures every scored token has at least 256 tokens of left context. KLD direction: KL(P_base ‖ P_quant) — "how well does the quantized model approximate the base?"

Same top-p = 92.6% means both quantized and base models agree on the most likely token 92.6% of the time.

Note on API-based KLD: These measurements use vLLM's top-20 logprobs per token (API limit), not full-vocab logits. This underestimates absolute KLD by ~10–15% compared to llama.cpp's full-vocab computation (see mlx-kld analysis). The corrected full-vocab KLD is estimated at ~0.09. Relative comparisons between quantization methods remain valid regardless.

KL Divergence vs BF16 (Generation / Autoregressive)

KLD measured per generation step during autoregressive decoding. Step-0 = first generated token (comparable to static/prefill KLD). Later steps compound — small per-token divergences accumulate as the two models diverge onto different token trajectories. This is normal and expected for 4-bit quantization.

Step-0 KLD (0.072–0.084) is consistent with the static prefill KLD (0.0746), with variance driven by prompt length. Shorter prompts have less context → more sensitive to quantization noise.

vLLM Throughput (RTX 5090, 32 GB)

Single Request

32 Concurrent Requests

Each request generated up to 1,024 tokens. Average latency includes queuing time under concurrent load.

Hardware Requirements

Minimum: 1× GPU with ≥24 GB VRAM (with fp8 KV cache and reduced context).

AutoRound W4A16 works on all GPU architectures (sm75+): Ampere (A100, RTX 3090), Ada (RTX 4090), Blackwell (RTX 5090, RTX PRO 6000). No architecture-specific tensor cores required — dequantization is handled by Marlin/CUTLASS kernels.

Usage with vLLM

Tested with: vllm/vllm-openai:cu130-nightly-fe9c3d6c5f66c873d196800384ed6880687b9e52 (vLLM v0.19.2rc1.dev134)

Docker Deployment

docker run -d --name vllm-mistral-autoround \
  --runtime=nvidia --gpus '"device=0"' \
  -p 8000:8000 \
  -v /path/to/model:/workspace/model \
  -v /path/to/vllm_config.yaml:/vllm_config.yaml \
  --ipc=host --shm-size=16g \
  --restart unless-stopped \
  vllm/vllm-openai:cu130-nightly-fe9c3d6c5f66c873d196800384ed6880687b9e52 \
  --config /vllm_config.yaml

Example vLLM Configuration (YAML)

This configuration is deployed and verified on an RTX 5090 (32 GB):

# -- Model & Server ----------------------------------------------------------
model: /workspace/model
host: "0.0.0.0"
port: 8000
served_model_name: "mistral-small-3.2-24b-Instruct-AWQ-W4A16"
trust_remote_code: true
tensor_parallel_size: 1

# -- Quantization ------------------------------------------------------------
quantization: auto_round

# -- Tokenizer & Config Format -----------------------------------------------
tokenizer_mode: mistral
config_format: mistral

# -- Data Type ---------------------------------------------------------------
dtype: bfloat16

# -- Load Format -------------------------------------------------------------
load_format: auto

# -- Context & Batching ------------------------------------------------------
max_model_len: 131072
max_num_batched_tokens: 8192
max_num_seqs: 32

# -- Memory ------------------------------------------------------------------
gpu_memory_utilization: 0.96
enable_prefix_caching: true
enable_chunked_prefill: true
kv_cache_dtype: fp8_e4m3

# -- Multi-Modal -------------------------------------------------------------
limit_mm_per_prompt:
  image: 4

# -- Tool Calling ------------------------------------------------------------
# Mistral native tool call format.
enable_auto_tool_choice: true
tool_call_parser: mistral

# -- Default Generation ------------------------------------------------------
generation_config: auto
override_generation_config:
  temperature: 0.15

# -- Misc --------------------------------------------------------------------
disable_custom_all_reduce: true

Inference Test

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"mistral-small-3.2-24b-Instruct-AWQ-W4A16","messages":[{"role":"user","content":"What is 2+2? One word."}],"max_tokens":10}'

Notes

Tokenizer: Tekken (Mistral-specific)

This model uses the Tekken tokenizer (tekken.json), not a standard HF tokenizer. You must use --tokenizer-mode mistral with vLLM to ensure correct tokenization. Using auto or hf mode produces garbled output.

Vision: Image Size Limit

The tekken.json in this repository has max_image_size set to 1024 (down from the original 1540). Images with any dimension exceeding 1024px are proportionally downscaled before vision encoding.

Files in This Repository

License

This quantization is released under the Apache 2.0 License, following the base model's license.

The base model mistralai/Mistral-Small-3.2-24B-Instruct-2506 is licensed under Apache 2.0.

Citation

If you use this model in your research, please cite:

@misc{mistral-small-3.2-24b-autoround-w4a16,
  title = {Mistral-Small-3.2-24B-Instruct-2506 AutoRound W4A16 Quantization},
  author = {Gratex International},
  year = {2026},
  howpublished = {\url{https://huggingface.co/gratex/Mistral-Small-3.2-24B-Instruct-2506-W4A16-AutoRound}},
  note = {Quantized with AutoRound 0.10.2}
}

Acknowledgments

This quantization was produced using hardware provided by Gratex International, a.s.

---

Original Model: mistralai/Mistral-Small-3.2-24B-Instruct-2506 Quantization Tool: AutoRound Deployment Engine: vLLM