X2-ASR-4B-0812

X2-ASR-4B-0812

Real-time bilingual streaming ASR (Chinese / English, including mixed speech).

X2-ASR-4B-0812 is Stage 1 of X2-Turn: a full fine-tune of mistralai/Voxtral-Mini-4B-Realtime-2602 on a large-scale aligned Chinese–English corpus. It keeps the original delayed-stream protocol (one token every 80 ms, configurable lookahead τ) and does not add a turn-taking head.

The dual-head model that jointly predicts ASR and turn state is x-square-robot/X2-Turn-4B-0812. Use that checkpoint for voice-assistant endpointing, barge-in, and the Turn Demo. Use this checkpoint when you only need streaming transcription, or as the backbone to train your own Stage 2.

Model summary

Do not load this checkpoint with load_mtp_checkpoint() from the X2-Turn package. That loader expects vad_lm_head, which this Stage 1 release does not have.

Quick start

This is a stock VoxtralRealtimeForConditionalGeneration checkpoint. No trust_remote_code is required.

pip install --upgrade transformers "mistral-common[audio]"

from transformers import AutoProcessor, VoxtralRealtimeForConditionalGeneration
from mistral_common.tokens.tokenizers.audio import Audio

model_id = "x-square-robot/X2-ASR-4B-0812"

processor = AutoProcessor.from_pretrained(model_id)
model = VoxtralRealtimeForConditionalGeneration.from_pretrained(
    model_id,
    dtype="bfloat16",
    device_map="auto",
).eval()

audio = Audio.from_file("/path/to/input.wav", strict=False)
audio.resample(processor.feature_extractor.sampling_rate)

inputs = processor(audio.audio_array, return_tensors="pt")
inputs = inputs.to(model.device, dtype=model.dtype)

outputs = model.generate(**inputs)
print(processor.batch_decode(outputs, skip_special_tokens=True)[0])

Default streaming delay is τ = 480 ms (6 × 80 ms), matching the pretrained Voxtral Realtime recipe. The same weights cover τ in 80–2400 ms; the tables below show the accuracy–latency trade-off.

A GPU with at least 16 GB of VRAM is recommended.

Relation to X2-Turn

Training follows the two-stage recipe in the paper:

1. Stage 1 (this checkpoint) — streaming ASR adaptation on ~26k hours of Chinese–English speech with frame-level ASR labels. Full fine-tune of the causal audio encoder and the language decoder. τ is sampled per batch between 1 and 30 frames (80–2400 ms).
2. Stage 2 — X2-Turn-4B-0812 attaches an independent turn head (initialized as a copy of the ASR lm_head) and jointly fine-tunes ASR + turn state.

Stage 2 is stronger at turn-taking. Stage 1 is stronger at ASR on the same streaming test sets (see below). After Stage 2 joint training, recognition degrades on some sets; that is expected.

Benchmark results

Streaming ASR of this checkpoint at three operating points of the delay τ. Chinese numbers are CER (%); English numbers are WER (%). Avg. is the unweighted mean over the listed sets. A subset of the 480 ms / 2400 ms numbers also appears in Table 3 of the paper, together with Stage 2 (X2-Turn) and chunk-based baselines.

Chinese (CER ↓)

English (WER ↓)

Larger τ uses more lookahead and generally lowers error, at the cost of additional streaming delay. AISHELL-4 and VoxPopuli Accented remain the hardest sets in each language. For Stage 2 turn-taking accuracy and the comparison against Uni-ASR / Freeze-Omni, see the paper.

Training data

Stage 1 uses public Chinese and English speech corpora totaling about 26k hours (about 14k Chinese / 12k English), with word-level timestamps from forced alignment:

• Chinese: AISHELL-1/2/3/4, AliMeeting, WenetSpeech, KeSpeech
• English: LibriSpeech, GigaSpeech, TED-LIUM, VoxPopuli

English words are tokenized with a leading space ( Hello / world), matching Voxtral’s BPE convention; CJK characters are unchanged.

Architecture

Unchanged from Voxtral-Mini-4B-Realtime-2602:

• Causal audio encoder + projector + language decoder
• Single ASR lm_head (no vad_lm_head)
• Delayed-stream tokens: [STREAMING_PAD] (id 32), [STREAMING_WORD] (id 33), then text
• Runtime metadata: params.json, tekken.json, processor_config.json, generation_config.json

The canonical model.safetensors is a single BF16 file.

Use cases

Recommended for:

• Low-latency Mandarin, English, and mixed-language streaming ASR
• Reproducing the paper’s Stage 1 ASR numbers
• Initializing a custom Stage 2 / turn-taking fine-tune

Not recommended for:

• Turn-taking, endpointing, backchannel detection, or barge-in — use X2-Turn-4B-0812
• Safety-critical decisions without human oversight
• Legal transcription as the sole basis for action

Limitations

• Accuracy may degrade under noise, reverberation, overlapping speakers, accents, dialects, code-switching, far-field microphones, or packet loss.
• This checkpoint does not predict turn state. Do not treat ASR completeness as a turn-end signal.
• Real-time latency depends on hardware, serving configuration, and the chosen τ.

Speech may contain personal, biometric, confidential, or copyrighted information. Obtain appropriate consent, minimize collection and retention, and avoid logging raw audio or transcripts by default.

Citation

If you use this model, please cite:

@article{fu2026x2turn,
  title   = {X2-Turn: Frame-Synchronous Dual-Head Modeling for Joint Streaming ASR and Turn State Prediction},
  author  = {Kaiqi Fu and Rime Wen and Altman Lin and Shawn Qin and Roy Gan and Hao Wang and Qian Wang},
  journal = {arXiv preprint arXiv:2608.10878},
  year    = {2026},
  url     = {https://arxiv.org/abs/2608.10878}
}

License

Model weights are released under Apache-2.0. Use is also subject to the Mistral base model terms. See NOTICE for attribution details.