ultravox-mamaylm-12b-uk-v2

Ultravox-MamayLM-12B-UK v2 (multi-dataset)

Single-pass speech-language model for Ukrainian, built on top of INSAIT-Institute/MamayLM-Gemma-3-12B-IT-v1.0 with the Ultravox v0.6 architecture (Whisper-large-v3-turbo audio encoder + projector → frozen Gemma-3-12B LLM).

This is the v2 checkpoint (HF tag v2.0): trained on a broader UK + EN dataset mix for 14 400 steps. v2 closes the verbatim-ASR WER gap with the Whisper-large-v3-turbo cascade pipeline on Ukrainian read-aloud audio while keeping the single-pass latency advantage.

Headline result

Same 50-fixture Ukrainian benchmark, same MamayLM-12B backbone, same prompts:

v2 vs v1: paired Δ = −0.117 (95 % CI [−0.150, −0.085], Cohen's d = −0.98, n = 51 fixtures, v2 better on 38/51, 2 verbatim rounds per version). v2's verbatim WER is statistically indistinguishable from the Whisper cascade's at 3.4× faster TTFT.

Full bench artifacts (v1 and v2): roman4work/voice-bench-results (bench-20260501T141534Z for v1, bench-20260502T081341Z for v2).

Architecture

• Audio encoder: openai/whisper-large-v3-turbo (LoRA-adapted, r = 8, target k/v/q/o_proj)
• Projector: SwiGLU, stack_factor = 8, mid-LayerNorm
• Text backbone: INSAIT-Institute/MamayLM-Gemma-3-12B-IT-v1.0 (frozen during training, loaded automatically by the Ultravox model class — you do not need to download it separately, but you must have access)

This repository contains only the projector + Whisper-LoRA + tokenizer / processor files (~130 MB). The base text model is referenced by config.json (text_model_id) and fetched from HF Hub at load time.

Training data (mix)

EN data is anchored at low weight to prevent the projector from collapsing onto UK-specific audio statistics. FLEURS contributes only its -transcription form because the dataset registry does not provide a -continuation version for FLEURS.

Training setup

Inference

Recommended runtime: vLLM with the vllm[audio] extras. Example deployment:

vllm serve roman4work/ultravox-mamaylm-12b-uk-v2 \
    --served-model-name ultravox-mamaylm-uk \
    --max-model-len 4096 \
    --dtype bfloat16 \
    --trust-remote-code \
    --enforce-eager \
    --block-size 64

The OpenAI-compatible Chat Completions endpoint accepts audio via the input_audio content type:

{
  "model": "ultravox-mamaylm-uk",
  "messages": [
    {"role": "system",
     "content": "Ти український голосовий помічник. Відповідай коротко, природно і виключно українською мовою."},
    {"role": "user", "content": [
      {"type": "input_audio",
       "input_audio": {"data": "<base64-encoded-wav>", "format": "wav"}}
    ]}
  ]
}

License

This model inherits from its components and you must comply with all of:

• Gemma Terms of Use (the MamayLM backbone is a Gemma-3 derivative)
• INSAIT MamayLM license
• Whisper LoRA weights are released under MIT (audio encoder)

Citation

If you use this checkpoint, please credit:

• Ultravox: Fixie AI Ultravox
• MamayLM: INSAIT-Institute/MamayLM-Gemma-3-12B-IT-v1.0
• Gemma: Gemma 3 Technical Report