Voxtral-Small-24B-2507-W4A16

Voxtral-Small-24B-2507-W4A16-GPTQ

Model Overview

• Model Architecture: VoxtralForConditionalGeneration
  • Input: Audio-Text
  • Output: Text
• Model Optimizations:
  • Weight quantization: INT4 (4-bit, GPTQ)
  • Activation quantization: FP16 (16-bit, unquantized)
• Quantization Scheme: W4A16 GPTQ
• Calibration Dataset: C4 (English) — 256 samples, max 2048 tokens
• Release Date: 03/27/2026
• Version: 1.0
• Model Developers: ghecko

Description

Quantized version of Voxtral-Small-24B-2507 using W4A16 GPTQ quantization via llmcompressor.

4-bit weight quantization significantly reduces model size and memory requirements while preserving audio understanding, transcription, and translation capabilities.

Quantization Details

• Method: GPTQ (one-shot, weight-only)
• Scheme: W4A16 — 4-bit integer weights, 16-bit floating-point activations
• Dampening: 0.1
• Ignored layers: lm_head, audio tower, multimodal projector (kept in original precision)

Usage

With Transformers

import torch
from transformers import VoxtralForConditionalGeneration, AutoProcessor

model_id = "ghecko78/Voxtral-Small-24B-2507-W4A16-GPTQ"

model = VoxtralForConditionalGeneration.from_pretrained(
    model_id,
    device_map="auto",
    dtype=torch.float16,
)
processor = AutoProcessor.from_pretrained(model_id)

With vLLM

from vllm import LLM

model = LLM(
    model="ghecko78/Voxtral-Small-24B-2507-W4A16-GPTQ",
    max_model_len=4096,
)

Creation

Created using the following quantization recipe:

from llmcompressor.modifiers.quantization import GPTQModifier

recipe = GPTQModifier(
    targets="Linear",
    scheme="W4A16",
    ignore=["language_model.lm_head", "re:audio_tower.*", "re:multi_modal_projector.*"],
    dampening_frac=0.1,
)

See the full quantization script: scripts/quantize.py