crisper-whisper2-unity

CrisperWhisperKo (Unity Sentis demo)

On-device Korean verbatim speech-to-text, running entirely in Unity with Sentis (com.unity.ai.inference) 2.6. Hold a button, speak Korean, release, and the clip is transcribed on-device with no server. "Verbatim" means fillers and repeats (어, 음, 그, 막 ...) are kept, not cleaned away.

What it does

• Press and hold the Hold to Record button to capture microphone audio (16 kHz).
• On release, the recording is transcribed and the Korean text is shown.
• Model size (Small / Base / Tiny) is selected in the Inspector (no runtime UI).

Pipeline

Microphone PCM (16 kHz, up to 30 s)
  -> log-mel graph      logmel.sentis        Conv1d/DFT, matches the HF Whisper feature extractor
  -> Whisper encoder    encoder.sentis       Float16
  -> Whisper decoder    decoder(_with_past)  Float16, manual KV cache
  -> Sentis tokenizer   -> Korean text

All three sizes share one log-mel graph. Everything loads from StreamingAssets/ko as .sentis (no raw ONNX at runtime).

Requirements

Unity 6000.5.5f1, com.unity.ai.inference (Sentis) 2.6.1, and a microphone (the OS may prompt for record permission on first use).

Run

1. Open this project in Unity.
2. Open Assets/Scenes/CrisperWhisperKo.unity and press Play.
3. Select the CrisperWhisperKo object in the Hierarchy, pick Model (Small / Base / Tiny) in the Inspector.
4. Hold the button, speak Korean, release to transcribe. Model can be switched live in the Inspector.

Project layout

Assets/
  Scenes/CrisperWhisperKo.unity     demo scene (UGUI: record button + output/status text)
  Scripts/
    CrisperWhisperKo.cs             mic capture + model select + transcription (manager)
    WhisperRunner.cs             encoder + manual-KV greedy decode
    WhisperText.cs               Sentis tokenizer decode
    WhisperConfig.cs             meta types
  Settings/                      URP asset + renderer
  StreamingAssets/ko/            models the demo loads
    logmel.sentis
    small/  base/  tiny/         encoder + decoder + decoder_with_past (Float16) + meta + tokenizer

The .sentis files are prebuilt (Float16 conversions of the Hugging Face models below). The offline conversion/quantization scripts are not part of this demo project.

Models

The .sentis files under StreamingAssets/ko are Float16 conversions of the CrisperWhisper2 Korean verbatim models. Full model cards, weights (safetensors), evaluation, and transcription examples are on Hugging Face:

How they were made. small is initialized from CrisperWhisper2.0_small and fine-tuned on Korean verbatim data (KsponSpeech ground truth); base and tiny are soft-KD distilled from that Korean small. This transfers CrisperWhisper's verbatim strength (an English-only model) into Korean. On held-out KsponSpeech, base (74M) beats standard whisper-small (244M, CER 0.171) at one third the size, and all sizes keep 86 to 92% of disfluencies (standard Whisper keeps about 60%).

License: non-commercial / research use only. The models inherit the CrisperWhisper2.0 Non-Commercial Research License (used as init / KD teacher / tokenizer) and the AI Hub KsponSpeech terms (training data). See each model card's NOTICE for the full license chain.