voxtral-streaming-asr

Voxtral Streaming ASR

VSA is a production-oriented Python client, frontend, benchmark, and evaluation stack for mistralai/Voxtral-Mini-4B-Realtime-2602 served by vLLM. It keeps file replay, microphone capture, protocol scheduling, latency measurement, and paired correctness reporting on one reusable realtime session core.

Frozen Runtime Identity

config.json is the production-equivalent client profile. The server/model identity is evidence, not an endpoint secret; connection host and port remain operator-supplied or loopback defaults.

Evidence Modes

VSA labels timing by origin:

• paced_websocket_replay: frozen WAV samples are sent at source speed through a real WebSocket session. This is the corpus qualification mode.
• capture_origin: microphone PCM carries one process-local time.monotonic() capture timestamp shared with output observations.
• unpaced_diagnostic: future WAV samples are sent immediately. Throughput is valid, but causal speech-to-partial latency is unavailable.
• deterministic_replay: no provider wall clock is mixed into causal latency.

Only capture_origin microphone evidence is true live. Transcript output and WAV capture are opt-in because they contain private speech.

Install

conda activate voxtral-streaming-asr
pip install -e '.[dev,file]'

The file extra is required only for resampling audio whose native rate is not 16 kHz.

Server

The qualified server is the vLLM identity above with the delay-240 tokenizer overlay and a server-side transcription delay of 240 ms. Keep the provider WebSocket at /v1/realtime. The repository does not provision models, tunnels, credentials, or public endpoints.

Quick Start

Stream one WAV at source speed:

python realtime_test.py sample.wav 127.0.0.1 18765 \
  --config config.json \
  --first-speech-s 0.0

Publish transcript partials as they arrive and then the final transcript:

python realtime_test.py sample.wav \
  --config config.json \
  --print-partials \
  --print-transcript

Run one bounded microphone session:

python realtime_test_mic.py \
  --config config.json \
  --duration-s 10 \
  --print-partials

Private raw microphone capture requires an explicit destination:

python realtime_test_mic.py \
  --config config.json \
  --duration-s 10 \
  --wav-out artifacts/private/mic-session.wav

Compare presets on one file:

python benchmarks/run_latency_sweep.py \
  --audio sample.wav \
  --preset production=config.json \
  --preset safe=configs/preset_safe_continuous.json \
  --repeat 3 \
  --output artifacts/latency-sweep.csv

Frozen-Corpus Evaluation

Set paths from private deployment configuration rather than committing them:

export SST_DATASET_ROOT=/path/to/generalizable-sst-dataset-v1
export KFOLD="$SST_DATASET_ROOT/eval-inputs/<frozen-accepted520-directory>"
export WAV_DIR="$SST_DATASET_ROOT/prepared/<frozen-release>/release/wav"
export CASE_PARTITIONS=/path/to/accepted520-case-partitions.json

Preflight all identities without network:

python scripts/evaluate_live_asr.py \
  --metadata "$KFOLD/metadata.csv" \
  --wav-dir "$WAV_DIR" \
  --fold-assignment "$KFOLD/fold_assignment.json" \
  --case-partitions "$CASE_PARTITIONS" \
  --config config.json \
  --output-dir artifacts/runs/accepted520-preflight \
  --preflight-only

Run development and validation before freezing a candidate:

python scripts/evaluate_live_asr.py \
  --metadata "$KFOLD/metadata.csv" \
  --wav-dir "$WAV_DIR" \
  --fold-assignment "$KFOLD/fold_assignment.json" \
  --case-partitions "$CASE_PARTITIONS" \
  --config config.json \
  --partitions development validation \
  --output-dir artifacts/runs/candidate-devval

After freezing the candidate, evaluate testing exactly once:

python scripts/evaluate_live_asr.py \
  --metadata "$KFOLD/metadata.csv" \
  --wav-dir "$WAV_DIR" \
  --fold-assignment "$KFOLD/fold_assignment.json" \
  --case-partitions "$CASE_PARTITIONS" \
  --config config.json \
  --partitions testing \
  --output-dir artifacts/runs/candidate-testing

The evaluator writes append-only attempts, a canonical latest-attempt projection, sanitized JSONL/CSV metrics, and a private private_case_outputs.jsonl containing transcript text. Generated evidence under artifacts/ is ignored.

Paired Comparison

scripts/compare_asr_runs.py projects SST and Nemotron source rows onto the exact candidate case IDs. SST sources may contain a larger source corpus; extra baseline rows are counted but never averaged into the paired result.

python scripts/compare_asr_runs.py \
  --candidate artifacts/runs/candidate-devval/cases.jsonl \
               artifacts/runs/candidate-testing/cases.jsonl \
  --sst "$SST_DATASET_ROOT/asr-quality/<frozen-run>/runtime_asr_matrix_cases.csv" \
  --nemotron "$NSAM_REPO/logs/<frozen-source-latency>/encoder_step_560ms/cases.jsonl" \
             "$NSAM_REPO/logs/<frozen-dev>/encoder_step_560ms/cases.jsonl" \
             "$NSAM_REPO/logs/<frozen-validation>/encoder_step_560ms/cases.jsonl" \
             "$NSAM_REPO/logs/<frozen-testing>/encoder_step_560ms/cases.jsonl" \
  --out-json artifacts/reports/paired-comparison.json \
  --out-md artifacts/reports/paired-comparison.md

The output preserves raw and normalized WER/CER, S/D/I, macro rates, exact match, partition/fold summaries, latency distributions, RTF, finalization tail, missing counts, and explicit gates. Missing metrics remain null.

Performance Summary

VSA has complete 520/520 timing and correctness coverage and is more accurate than both references. The requested aggregate p99 objective passes at 1.482097 s linear interpolation and 1.489568 s nearest rank. Full qualification still fails: the raw first-partial maximum is 3.500405 s, and its better-than-SST micro deltas (0.368 pp WER, 0.321 pp CER) exceed the preregistered 0.20 pp compatibility band. Commit cadence, append granularity, and safe continuous preprocessing do not remove the provider’s semantic-readiness tail. See docs/performance.md for the full comparison contract and current qualification status.

Configuration

Silence gating is disabled by default. Frontend details and artifact checks are in docs/frontend.md.

Development

python scripts/quality_gate.py --coverage-min 90

The gate runs Ruff, deterministic tests, aggregate coverage, and per-file coverage. Ordinary tests use synthetic PCM and fake transports; they require no microphone, GPU, private audio, or network.

Additional references:

• ARCHITECTURE.md: runtime and protocol ownership.
• docs/live-input.md: file, replay, and microphone clocks.
• docs/evaluation.md: corpus and paired metric contracts.
• docs/performance.md: frozen baselines and gate status.

License

Apache License 2.0. See LICENSE and NOTICE.