kupe-asr-en

<img src="https://huggingface.co/anuj-inavlabs/kupe-asr-en/resolve/main/kupe-light.png" height="44" alt="logo"/>

Streaming English ASR — early Phase‑1 checkpoint

A 670M streaming speech‑to‑text model: audio is turned into Mimi codec tokens and the Kupe‑SLM‑670M language backbone reads them and writes English text. Small, fast, real‑time.

⚠️ Early Phase‑1 checkpoint. This is a first sanity‑milestone model trained on ~2.6k h to prove the pipeline and architecture. It is not the final release — quality will improve substantially with more data, augmentation, and epochs in later phases.

---

Overview

• Task: English automatic speech recognition (streaming + full‑file).
• How it works: speech → Mimi neural audio codec (8 codebooks @ 12.5 Hz) → the 8 codebooks are summed into one embedding per 80 ms frame → fed to Kupe‑SLM‑670M as inputs_embeds → the model decodes English text. No separate acoustic encoder.
• Streaming: emits turn signals from the model's own end‑of‑sequence probability — PRE_HIT_LLM (prefetch your LLM) and END_OF_SPEECH (commit final).
• Anti‑hallucination: trained with 5 % silence/ambient clips labelled "", so silence decodes to nothing (measured 100 %).

Results in context

Production English ASR on much larger corpora sits around 4–6 % WER. 10.3 % from a Phase‑1 670M model on 2.6k h is a strong early result and already usable for voice agents. Live WER during training fell 41 % → 18 % → 12.8 % → 10.3 %.

Training data (~2.6k h English, encoded)

24 kHz mono, Mimi‑encoded (8 codebooks). Splits: train ~2,733 h · val 5.1 h · test 5.0 h.

Training setup (1× H100 80 GB)

Tokenizer efficiency (fertility)

The 131,072‑vocabulary tokenizer is very efficient — low fertility (tokens/word) means shorter targets, faster decoding, lower CER.

Multilingual outlook (this checkpoint is EN‑only)

The backbone was pre‑trained on English + 11 Indic languages; a probe found it already generates coherent Hindi & Gujarati and has 100 % tokenizer round‑trip across all 11 scripts. So extending to Hindi/Gujarati/… is mainly a matter of more data on the same pipeline — the model side is ready.

Files

• kupe-lm/ — the Kupe‑SLM‑670M backbone (weights + tokenizer).
• frontend.pt — audio frontend (codebook embeddings + projector).
• asr_config.json — frontend mode (per_frame_sum), codebooks (8), embed dim.

Inference

Requires transformers==5.4.0 and the kupe-asr-en code.

from kupe_asr_en.modeling.asr_model import LummaASR   # loader class name; loads Kupe-SLM
model, tok = LummaASR.load("anuj-inavlabs/kupe-asr-en", device="cuda")

Streaming from a wav (turn flags):

from kupe_asr_en.stream import transcribe_file
for ev in transcribe_file("model_dir", "audio.wav"):
    print(ev.type, ev.text)   # PARTIAL / PRE_HIT_LLM / END_OF_SPEECH

Live mic / Gradio demo (file + realtime, with RTF):

python scripts/06_mic_stream.py --model-dir <this repo>     # terminal mic
python scripts/10_gradio_kaggle.py                          # web UI (file + live mic)

Related repos

• Data: anuj-inavlabs/kupe-asr-en-data
• Runs: anuj-inavlabs/kupe-asr-en-runs
• Code: https://github.com/iNavLabsResearch/kupe-asr-en

License & credits

Speech‑LM backbone weights are used under their upstream license; audio codec: kyutai/mimi. Data: LibriSpeech, MLCommons People's Speech (CC‑BY), AI4Bharat Svarah. Released as other. Built by iNavLabs.