mistral-small-3.2-24b-Instruct-2506-NVFP4A16

Mistral-Small-3.2-24B-Instruct-2506 — NVFP4A16 Quantization

This is a NVFP4A16 (NVIDIA FP4 weight-only) quantization of Mistral-Small-3.2-24B-Instruct-2506, a 24B parameter multimodal model with Pixtral-style vision capabilities, quantized using LLM Compressor v0.10.0.2 with NVFP4A16 scheme via post-training quantization (PTQ) oneshot.

NVFP4A16 stores weights in NVIDIA's native FP4 format (4-bit floating-point) with BF16 activations. On Blackwell GPUs (RTX 5090, RTX PRO 6000, B200), this format leverages dedicated FP4 tensor cores that should deliver ~1.3× faster inference compared to INT4 weight-only formats (GPTQ, AWQ, AutoRound) that must dequantize via Marlin/CUTLASS before matrix multiply.

Model Details

Quantization Configuration

{
  "config_groups": {
    "group_0": {
      "format": "nvfp4-pack-quantized",
      "input_activations": null,
      "output_activations": null,
      "targets": ["Linear"],
      "weights": {
        "num_bits": 4,
        "type": "float",
        "group_size": 16,
        "strategy": "tensor_group",
        "symmetric": true,
        "observer": "memoryless_minmax",
        "scale_dtype": "torch.float8_e4m3fn",
        "dynamic": false
      }
    }
  },
  "format": "nvfp4-pack-quantized",
  "quant_method": "compressed-tensors",
  "quantization_status": "compressed"
}

Key parameters:

• scheme=NVFP4A16: Weight-only FP4 quantization — no activation quantization. Critical for VLMs: activation quantization destroys vision quality.
• group_size=16: Per-16-element scaling groups (NVFP4 native granularity)
• strategy=tensor_group: One scale per tensor group (NVIDIA's recommended strategy for FP4)
• scale_dtype=float8_e4m3fn: Scales stored in FP8 E4M3 for hardware efficiency
• symmetric=true: Symmetric quantization (no zero-point)
• observer=memoryless_minmax: Min/max calibration without temporal smoothing

Recipe

default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore: [lm_head, 're:.*vision_tower.*', 're:.*multi_modal_projector.*']
      scheme: NVFP4A16
      bypass_divisibility_checks: false

Calibration Dataset

The model was calibrated using a domain-specific mix of text-only datasets (no images/video, to avoid torchvision import errors). Short samples were concatenated into chunks of ≥2,048 Tekken tokens each:

Quality Benchmarks

All benchmarks use wikitext-2-raw-v1 (test split) — the standard dataset for quantization quality evaluation, matching the methodology of llama.cpp, AutoAWQ, GPTQ, and the academic literature.

WikiText-2 Perplexity (ctx=512)

Measured via vLLM completions API with echo=True + logprobs. 642 non-overlapping chunks, ~301K tokens scored. Methodology matches llama.cpp ./perplexity -c 512.

KL Divergence vs BF16 (Static / Prefill)

KL divergence measures how much the output probability distribution has shifted from the base model. Lower is better; 0 = identical.

Methodology matches llama.cpp --kl-divergence: wikitext-2-raw-v1, ctx=512, score only the second half of each chunk (positions [256–511]), which ensures every scored token has at least 256 tokens of left context. 32 chunks, ~8,150 tokens scored. KLD direction: KL(P_base ‖ P_quant) — "how well does the quantized model approximate the base?"

Same top-p = 91.1% means both quantized and base models agree on the most likely token 91.1% of the time.

Note on API-based KLD: These measurements use vLLM's top-20 logprobs per token (API limit), not full-vocab logits. This underestimates absolute KLD by ~10–15% compared to llama.cpp's full-vocab computation (see mlx-kld analysis). The corrected full-vocab KLD is estimated at ~0.10–0.11. Relative comparisons between quantization methods remain valid regardless.

KL Divergence vs BF16 (Generation / Autoregressive)

KLD measured per generation step during autoregressive decoding. Step-0 = first generated token (comparable to static/prefill KLD). Later steps compound — small per-token divergences accumulate as the two models diverge onto different token trajectories. This is normal and expected for 4-bit quantization.

Step-0 KLD (0.088–0.140) is consistent with the static prefill KLD (0.0935), with variance driven by prompt length. Shorter prompts have less context → more sensitive to quantization noise. At 512-token prompts, KLD drops to 0.088, closely matching the AutoRound W4A16's static KLD (0.0746).

vLLM Throughput (RTX 5090, 32 GB)

Single Request

32 Concurrent Requests

Each request generated up to 1,024 tokens. Average latency includes queuing time under concurrent load.

Hardware Requirements

Minimum: 1× GPU with ≥24 GB VRAM (with fp8 KV cache and reduced context).

NVFP4A16 requires Blackwell or later (sm120+): RTX 5090, RTX PRO 6000, B200 (with driver 580+). On Ampere/Ada GPUs (sm80–sm89), vLLM will fall back to software dequantization — NVFP4A16's speed advantage disappears.

Usage with vLLM

Tested with: vllm/vllm-openai:cu130-nightly-fe9c3d6c5f66c873d196800384ed6880687b9e52 (vLLM v0.19.2rc1.dev134)

Docker Deployment

docker run -d --name vllm-mistral-nvfp4 \
  --runtime=nvidia --gpus '"device=0"' \
  -p 8000:8000 \
  -v /path/to/model:/workspace/model \
  --ipc=host --shm-size=16g \
  vllm/vllm-openai:cu130-nightly-fe9c3d6c5f66c873d196800384ed6880687b9e52 \
  /workspace/model \
  --host 0.0.0.0 --port 8000 \
  --tokenizer-mode mistral \
  --config-format hf \
  --load-format auto \
  --quantization compressed-tensors \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.96 \
  --max-model-len 131072 \
  --disable-custom-all-reduce \
  --trust-remote-code \
  --limit-mm-per-prompt '{"image":4}' \
  --tool-call-parser mistral \
  --enable-auto-tool-choice

Example vLLM Configuration (YAML)

model: /workspace/model
quantization: compressed-tensors
tokenizer_mode: mistral
config_format: hf
load_format: auto
dtype: bfloat16
gpu_memory_utilization: 0.85
max_model_len: 32768
max_num_batched_tokens: 8192
kv_cache_dtype: fp8_e4m3
enable_prefix_caching: true
enable_chunked_prefill: true
limit_mm_per_prompt:
  image: 4
enable_auto_tool_choice: true
tool_call_parser: mistral
disable_custom_all_reduce: true

Inference Test

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"/workspace/model","messages":[{"role":"user","content":"What is 2+2? One word."}],"max_tokens":10}'

Notes

Tokenizer: Tekken (Mistral-specific)

This model uses the Tekken tokenizer (tekken.json), not a standard HF tokenizer. You must use --tokenizer-mode mistral with vLLM to ensure correct tokenization. Using auto or hf mode produces garbled output.

Vision: Image Size Limit

The tekken.json in this repository has max_image_size set to 1024 (down from the original 1540). Images with any dimension exceeding 1024px are proportionally downscaled before vision encoding.

Single-Shard Structure

This model is stored as a single model.safetensors file (~15 GB), containing all 40 quantized LM decoder layers + vision tower (BF16) + projector (BF16) + lm_head (BF16) + embed_tokens (BF16). No sharding.

Files in This Repository

License

This quantization is released under the Apache 2.0 License, following the base model's license.

The base model mistralai/Mistral-Small-3.2-24B-Instruct-2506 is licensed under Apache 2.0.

Citation

If you use this model in your research, please cite:

@misc{mistral-small-3.2-24b-nvfp4a16,
  title = {Mistral-Small-3.2-24B-Instruct-2506 NVFP4A16 Quantization},
  author = {Gratex International},
  year = {2026},
  howpublished = {\url{https://huggingface.co/gratex/Mistral-Small-3.2-24B-Instruct-2506-NVFP4A16}},
  note = {Quantized with LLM Compressor 0.14.1}
}

Acknowledgments

This quantization was produced using hardware provided by Gratex International, a.s.

---

Original Model: mistralai/Mistral-Small-3.2-24B-Instruct-2506 Quantization Tool: LLM Compressor Deployment Engine: vLLM