Voxtral-Mini-3B-2507-8bit-dense-encoder

Voxtral-Mini-3B-2507, 8-bit with a bf16 audio encoder (MLX)

Voxtral Mini quantised for Apple Silicon, with one deliberate asymmetry: the audio encoder and its projector stay in bf16 while the language model and lm_head are quantised to 8 bit.

Used in noScribe

These weights are one of the two transcription models in noScribe — a free, open-source app for high-quality interview transcripts (speaker diarization, timestamps, editable output). This is the recommended of the two: it runs on any Mac with 16 GB, and on hard conversational audio it is more accurate than the 24B Small build.

noScribe's optional Voxtral engine (Apple Silicon / MLX) lists it as voxtral-mini-8bit and downloads it on first use, so nothing here has to be fetched by hand. Setup and the engine's own notes: VOXTRAL.md. The engine currently lives on the voxtral branch of this fork of kaixxx/noScribe.

Why the encoder is left dense

The encoder runs once per pass; lm_head runs once per generated token. So encoder precision buys accuracy at almost no cost in speed, while lm_head precision costs throughput and — measured — buys nothing.

Sweeping the encoder alone on the 24B sibling (everything else held fixed) on hard German conversational audio:

Monotone, with the character error rate halving between 6 and 8 bit. Above 8 bit nothing more is gained, which is why "8 bit or dense" is the right target and 4-bit encoders are a bad idea.

The same sweep on lm_head (6 / 8 / bf16) produced identical error rates while bf16 cost 27 % throughput.

Measured quality

Against a hand-corrected reference of hard German audio — two speakers, some crosstalk, brand names, 422 words:

On that passage this build is byte-identical to bf16 at 4.5x the speed.

For context, on the same audio and the same metric: whisper-large-v3-turbo (via faster-whisper, int8, beam 5) scored 14.22 % WER, with 34 insertions including 27 consecutive words of hallucinated text.

Honest limits

• The advantage only shows on difficult audio. On FLEURS German (100 recordings, read-aloud, clean) this build and the uniform 8-bit one score identically: 4.81 % WER. If your material is clean, save the 0.6 GB.
• The hard-audio result rests on a single 422-word passage. The direction is consistent with the encoder sweep above, but a 0.5-point difference on 422 words is four words. Treat it as suggestive, not settled.
• Measured on an M1 Max with mlx-voxtral 0.0.4. Other MLX versions and machines will differ.
• On FLEURS the 24B model is clearly better than any 3B build (2.82 %); on the hard passage the 3B build wins (4.27 % vs 7.82 %). Pick by material, not by parameter count.

Usage

from mlx_voxtral import load_voxtral_model, VoxtralProcessor
import mlx.core as mx, soundfile as sf

model, _ = load_voxtral_model("MarkusKaemmerer/Voxtral-Mini-3B-2507-8bit-dense-encoder",
                              dtype=mx.bfloat16)
proc = VoxtralProcessor.from_pretrained("MarkusKaemmerer/Voxtral-Mini-3B-2507-8bit-dense-encoder")

audio, sr = sf.read("speech.wav", dtype="float32")   # 16 kHz mono
inp = proc.apply_transcrition_request(audio=audio, language="de", sampling_rate=sr)
out = model.generate(input_ids=inp.input_ids, input_features=inp.input_features,
                     max_new_tokens=4096, temperature=0.0, repetition_penalty=1.0)
print(proc.decode(out[0, inp.input_ids.shape[1]:], skip_special_tokens=True))

Set repetition_penalty=1.0. mlx-voxtral defaults to 1.2, a chat default that in verbatim speech penalises punctuation and function words: measured on a 10-minute German podcast it cost 27 % of all commas and swallowed real repetitions ("sehr, sehr" becoming "sehr").

Reproducing

Built with the conversion script from this noScribe fork:

python tools/quantize_voxtral.py mistralai/Voxtral-Mini-3B-2507 \
    out-dir 8 64 dense-encoder --lm-head-bits 8

The measurement scripts (wer.py, fleurs.py) live in the same repository under docs/skripte/, and the full write-up is in docs/voxtral-quantisierung.md.

License

Apache 2.0, inherited from mistralai/Voxtral-Mini-3B-2507.