granite-speech-5.0-470m-turboctc

granite-speech-5.0-470m-turboctc — LiteRT

ibm-granite/granite-speech-5.0-470m-turboctc converted to LiteRT (.tflite) for on-device speech recognition. A 473M CTC conformer: one encoder forward, no decoder loop, no KV cache — audio features in, token ids out, and the whole "decoder" on the host is ten lines of CTC collapse plus a tokenizer.

Verified on a Galaxy S26 (SM-S942Q) with real inputs: transcripts from both files match the fp32 PyTorch model exactly on every signature, and the int8 logits are bit-identical between Apple M4 Max and the S26 (max abs diff 0.0 — the dynamic int8 path is deterministic across devices). Use int8 on device: fp16 is ~3× slower on CPU because XNNPACK repacks fp16 weights to fp32.

This is a CPU model. The mobile GPU delegate does not compile the graph (measured on S26: Failed to compile model; the relative-position gather lowers to GATHER_ND, which mobile GPUs decline) — that is expected, not a defect, and CPU is faster than real-time by a wide margin anyway.

Signatures

Three fixed audio windows, batch 1. Input input_features float32 [1, 50×s, 320]; outputs ids int32 [1, T] (argmax over the CTC vocab, already computed in-graph) and logits float32 [1, T, 16384].

Zero-pad the 16 kHz waveform to the smallest window that fits and run that signature; trailing silence decodes to CTC blanks. The encoder's attention is block-local (128 frames ≈ 2.56 s), so padding can shift block boundaries: on 20 LibriSpeech clips, 19 transcribed identically to the unpadded model and one changed a single word. Longer audio: chunk at pauses and concatenate transcripts.

Front-end (host side)

16 kHz mono float32 → log-mel → normalize → delta → stack:

1. mel spectrogram: n_fft 512, win_length 400, hop_length 160, 80 mels (torchaudio MelSpectrogram defaults otherwise);
2. log10 (clip at 1e-10), then floor at max − 8.0 per utterance, then x/4 + 1;
3. append deltas (torchaudio.functional.compute_deltas, win 3) → 160 channels;
4. stack every 2 frames → [T', 320], T' = mel_frames / 2.

The usage snippet below is the exact contract, mirroring the upstream repo's processor.

Decode (host side)

Take the ids output, collapse repeats, drop blanks (id 0), decode with the bundled tokenizer.json. That's the entire decoder.

Usage (Python)

import numpy as np, torch, torchaudio
from ai_edge_litert.interpreter import Interpreter
from tokenizers import Tokenizer

it = Interpreter(model_path="granite_speech_ctc_wi8fc.tflite", num_threads=8)
tok = Tokenizer.from_file("tokenizer.json")
mel = torchaudio.transforms.MelSpectrogram(16000, n_fft=512, win_length=400,
                                           hop_length=160, n_mels=80)

def features(wave):                       # wave: float32 [T], 16 kHz mono
    x = torch.from_numpy(wave)[None]
    n_frames = 2 * -(-(x.shape[1] // 160) // 2)
    need = (n_frames - 1) * 160 + 1
    if x.shape[1] < need:
        x = torch.nn.functional.pad(x, (0, need - x.shape[1]))
    m = mel(x)[..., :n_frames].clip(min=1e-10).log10()
    m = torch.maximum(m, m.amax(dim=(-2, -1), keepdim=True) - 8.0) / 4 + 1
    m = torch.cat((m, torchaudio.functional.compute_deltas(m, win_length=3)), dim=-2)
    m = m.transpose(-2, -1).contiguous()
    return m.reshape(1, -1, 2 * m.shape[-1]).numpy()  # [1, T', 320]

def transcribe(wave):
    sec = next(s for s in (5, 10, 30) if len(wave) <= s * 16000)
    padded = np.zeros(sec * 16000, np.float32); padded[:len(wave)] = wave
    out = it.get_signature_runner(f"transcribe_{sec}s")(input_features=features(padded))
    ids = next(v for v in out.values() if v.dtype == np.int32)[0]
    collapsed, prev = [], -1
    for i in ids:
        if i != prev and i != 0:
            collapsed.append(int(i))
        prev = i
    return tok.decode(collapsed).strip()

wave, sr = torchaudio.load("speech.wav")          # resample to 16 kHz mono first
print(transcribe(wave[0].numpy()))

Quality (conversion parity)

On 20 LibriSpeech dev-clean clips (2–30 s): int8 and fp16 transcripts match the fp32 PyTorch model exactly, 20/20 each, and corpus WER is identical to fp32 under the same windows (4.24%; 3.79% unpadded — the gap is the fixed-window effect above, +2 words in 448). This is parity evidence on a small fixture set, not an ASR benchmark; for task quality see the base model card.

Speed (measured)

Latency per window (median; the model processes the whole window regardless of speech length, so this is also the worst case). Real fixture audio, outputs read every run.

Apple M4 Max (CPU/XNNPACK, ai_edge_litert Interpreter, 8 threads, median of 20 warm runs):

Galaxy S26, SM-S942Q (CPU, LiteRT CompiledModel via litert 2.2.0, median of 20 warm runs, thermal NONE→NONE):

Conversion

Encoder lane — a direct multi-signature litert_torch trace (no LLM export machinery). Two notes for reproducers:

• The Hub checkpoint and its bundled remote code disagree (as of 2026-09-01): the safetensors use the tensor names of the unreleased-transformers GraniteSpeech5ForCTC, while the shipped .py files expect the older packaged naming. The conversion loads through the remote code with a full-key state-dict rename (strict load, all 550 tensors).
• Shaw's re-association matters for export size. The rel-pos bias einsum(q, Emb(dists)) constant-folds under torch.export into a per-layer, per-block-length fp32 constant (~0.5 GB across signatures) that dynamic int8 can't reach. Rewriting it as gather(q @ W.T, dists) keeps one shared [1025, 128] table and is bitwise-identical in eager — int8 goes 923 → 518 MB.

Scripts and full notes: hf-to-litertlm.

License

Apache-2.0, inherited from the base model; LICENSE is included. (Note the separate -nc sibling of the base model is CC-BY-NC-SA — this conversion is of the Apache-2.0 release only.)

Modification notice: these files are converted, not original. The weights were exported to LiteRT and quantized (int8 dynamic-range / fp16); the in-graph argmax output was added. No fine-tuning or weight modification beyond quantization was performed.