voxtral_model

Voxtral Realtime 4B

Streaming speech-to-text model with ~4 billion parameters. Weights in BF16 safetensors format, extracted from mistralai/Voxtral-Mini-4B-Realtime-2602.

Architecture

Pipeline:

WAV → 16kHz → Mel Spectrogram → Conv Stem → Encoder → Downsample 4x → Adapter → Decoder → Tokens

• Audio Encoder: ~0.6B params — causal transformer, 32 layers
• Audio-Language Adapter: 2-layer MLP with 4x downsample
• LLM Decoder: ~3.4B params — Ministral-3 based, 26 layers with GQA

Audio Preprocessing

Encoder (Causal Transformer)

Conv stem: conv1d(128→1280, k=3, s=1) → GELU → conv1d(1280→1280, k=3, s=2) → GELU

Adapter

[seq/4, 5120] → Linear(5120→3072) → GELU → Linear(3072→3072) → [seq/4, 3072]

Decoder (LLM)

The decoder uses adaptive RMS normalization conditioned on transcription delay (6 delay tokens = 480ms).

Weight Format

• consolidated.safetensors (8.3 GB) — 711 tensors, all BF16
• params.json — model config
• tekken.json (14.9 MB) — Tekken tokenizer

Tokenizer (Tekken)

Token IDs 0–999 are special tokens. IDs 1000+ index into the vocabulary (base64-encoded byte sequences in tekken.json).

Audio Streaming Config

Decode Schedule (Offline)

1. Prompt: [BOS] + [STREAMING_PAD] × 38 (1 + 32 left-pad + 6 delay)
2. Prefill: Feed audio_embed[i] + tok_embed(prompt[i]) for positions 0..L-2
3. First token: Greedy argmax from position L-1
4. Autoregressive decode: For each remaining audio position, feed audio_embed[pos] + tok_embed(prev_token), greedy argmax
5. Stop: On EOS or end of audio span

C Implementation

A pure C implementation of this model is available at voxtral.c — runs on Apple Silicon (Metal) and CPU (BLAS), with streaming microphone input.

Credits

Original model by Mistral AI: mistralai/Voxtral-Mini-4B-Realtime-2602

Built for the Mistral Hackathon 2026.