Voxtral-Mini-3B-2507-AWQ4

Voxtral-Mini-3B-2507 AWQ4 (W4A16): EN, ARG and GEN

Three AWQ4 W4A16 quants of mistralai/Voxtral-Mini-3B-2507. They share the same recipe and differ only in the calibration data, so the comparison below isolates what calibration buys. Each variant lives in its own folder.

Note on parameter count. Hugging Face reports ~5B parameters from the safetensors metadata. The count is inflated because AWQ4 packs each group of 8 int4 weights into a single int32. Each variant's weights take about 4.36 GB on disk and in VRAM.

Benchmarks vs bf16 baseline

Data: FLEURS + Common Voice 15 + Multilingual LibriSpeech + ylacombe/google-argentinian-spanish (es-AR). Hardware: 2x RTX 4090, vLLM 0.25. WER in %, lower is better. Bold = best 4-bit variant per row.

All three variants halve the weight memory of bf16 and run about 9% faster. None of them dominates. The text-calibrated EN variant has the lowest six-language average, slightly below bf16 itself. Audio calibration on Spanish (ARG) gives the best Argentine Spanish and Dutch, but the worst French. Calibrating on all eight languages (GEN) helps es, it and pt, but costs on de, fr and nl. On Argentine Spanish the three variants are within 0.11 points of each other, and all are 0.3 to 0.4 points above bf16.

Which one to pick

• Mixed European languages: EN/.
• Argentine or LatAm Spanish: ARG/.
• Spanish, Italian or Portuguese: GEN/.

Usage (vLLM)

vLLM serves a local directory, so download only the variant you need:

# 1. In your vLLM venv, add the deps Voxtral needs
uv pip install --python /path/to/vllm/venv/python "mistral-common[soundfile]"

# 2. Download one variant (ARG here)
hf download juanginer/Voxtral-Mini-3B-2507-AWQ4 --include "ARG/*" --local-dir voxtral-awq4

# 3. Serve
vllm serve voxtral-awq4/ARG \
    --tokenizer-mode mistral \
    --config-format mistral \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.9 \
    --port 8000

Then send audio to POST /v1/audio/transcriptions in OpenAI-compatible format.

Quantization

All three use llmcompressor 0.10 with the same AWQ modifier: W4A16 scheme, group_size=128, duo_scaling=True, n_grid=20, lm_head left unquantized. It targets the Linear modules inside LlamaDecoderLayer. Each folder's recipe.yaml is the exact recipe used. The compressed-tensors format matches what vLLM 0.25.x expects for AWQ models, with weights re-fused for the vLLM Voxtral loader.

Calibration scales follow the activations they see. EN is tuned on text-token activations only. ARG and GEN pass their audio through the audio encoder, so their scales see real audio prefill activations.

Base model & license

Base: mistralai/Voxtral-Mini-3B-2507, Apache-2.0. These quantized derivatives are released under the same license.