com.sky.sentis.whisper

sentis-whisper

Whisper speech-to-text converted to Unity Inference Engine (Sentis) FP16. Both tiny and base sizes are included.

Files

logmel_spectrogram_fp16.sentis        # audio -> log-mel front end
vocab.json                            # token vocabulary for de-tokenization
tiny/   base/
  encoder_model_fp16.sentis           # mel -> encoder_hidden_states
  decoder_model_fp16.sentis           # first pass: tokens (+ encoder states) -> logits + present KV
  decoder_with_past_model_fp16.sentis # subsequent steps: next token + past KV -> logits + new KV
WhisperRecognizer.cs                  # self-contained Unity Sentis inference

Pipeline

This is a four-model greedy-decode pipeline (encoder KV is computed once; the decoder loop carries its own self-attention KV so cost is linear in tokens):

1. Log-mel — audio (16 kHz mono) → log-mel spectrogram tensor.
2. Encoder — mel → encoder_hidden_states (run once per utterance).
3. Decoder (prefill) — decoder_model over the prompt tokens (<|startoftranscript|>, language, <|transcribe|>, <|notimestamps|>) → first-token logits + present.* KV.
4. Decoder loop — decoder_with_past_model with the last token + past_key_values.* → next-token logits + updated KV. Argmax each step (greedy), append, repeat until <|endoftext|>.
5. De-tokenize the collected ids with vocab.json.

A complete self-contained implementation lives in WhisperRecognizer.cs (log-mel → encoder → incremental KV greedy decode, language auto-detect + lock, LocalAgreement-2 streaming partials, byte-level BPE de-tokenization, GPU warmup). Minimal usage:

var recognizer = new WhisperRecognizer(WhisperRecognizer.ModelSize.Tiny, BackendType.GPUCompute);
recognizer.Load(modelRoot); // folder holding vocab.json + logmel + tiny/ and base/ subfolders
recognizer.FinalTranscript += text => Debug.Log(text);
recognizer.BeginUtterance();
recognizer.PushSamples(samples, samples.Length, startSequence); // 16 kHz mono, monotonic sequence
recognizer.EndUtterance();

License & attribution

MIT. Converted from openai/whisper (MIT). The weights are derivatives of the upstream Whisper checkpoints; see the upstream license for the original terms.