stt-be-fastconformer-rnnt-coreml

Belarusian FastConformer RNN-T - Core ML (fp16)

fp16 Core ML (mlprogram) export of the RNN-T (Transducer) path of nvidia/stt_be_fastconformer_hybrid_large_pc

• a 115M-parameter FastConformer Transducer-CTC model for Belarusian speech recognition that emits punctuated, capitalized text. Runs fully on-device on Apple Silicon (CPU / GPU / Neural Engine), no Python and no NeMo at inference.

The model is split into three packages: the encoder runs once per 30 s audio window, the decoder and joint run in a host-side greedy loop. The log-mel front-end stays outside Core ML and is reproduced from the checkpoint's own window and filterbank buffers (mel_buffers.npz).

Files

The repo is self-contained: everything needed for on-device inference is here.

I/O contracts

Encoder: features float32 [1, 80, 2999] (log-mel, 30 s window, zero-padded right), length int32 [1] (true mel frames, so the padded tail is masked) -> encoded [1, 512, 375]. encoded valid length is computed host-side as L = (L-1)//2 + 1 applied three times to the mel length (8x downsampling); fp16 cannot represent the length exactly, so it is not returned by the graph.

Decoder step: token int32 [1, 1], h_in/c_in float32 [1, 1, 640] -> dec_out [1, 640], h_out, c_out. Start state: token = 1024 (blank), zero h/c - the blank embedding row is zeros (blank_as_pad), reproducing the reference fresh start exactly.

Joint step: enc_t [1, 512] (one encoder frame), dec_t [1, 640] -> logits [1, 1025]. blank_id = 1024; the 1024 non-blank ids index tokens.json.

Mel front-end (must match exactly): 16 kHz mono, preemphasis 0.97, n_fft = 512, win_length = 400, hop_length = 160, Hann window, center = True (reflect pad), 80 mel bins, power spectrum, log(x + 2^-24), then per-feature mean/variance normalization over the valid frames. The window and filterbank are embedded in mel_buffers.npz; the full upstream config is in model_config.yaml.

Usage

pip install coremltools torch numpy sentencepiece   # + ffmpeg on PATH
python example_infer.py audio.wav

example_infer.py loads the three packages, computes the mel from the embedded buffers, runs the encoder once per <=30 s window and a greedy RNN-T loop over the decoder + joint, and detokenizes with SentencePiece. No NeMo is required.

Longer audio: the example splits into <=30 s windows at low-energy boundaries and resets h/c/last per window. The emission frame index t maps to seconds as t * 8 * 160 / 16000 = t * 0.08 (8x downsampling, 10 ms hop) plus the window offset.

Conversion fidelity

Verified against the NeMo PyTorch reference (fp32 CPU) on three Belarusian FLEURS clips, decoding the same windows. The NeMo-free standalone mel is itself token-exact against NeMo's AudioToMelSpectrogramPreprocessor.

The ANE runs the encoder in higher-noise fp16 accumulation; as with other Conformer exports, a borderline low-confidence emission can flip to blank on the ANE. For transcripts that must match the reference exactly, use CPU_AND_GPU.

Latency

On Apple Silicon (CPU_AND_GPU, single clip) the full pipeline runs at RTF ~0.006-0.01 (roughly 100-160x faster than real time); the greedy loop is not the bottleneck.

Base model quality

From the base model card: WER 2.72% without punctuation/capitalization and 3.87% with, on Common Voice 12 Belarusian test (Transducer decoder, greedy). These are clean read-speech numbers; expect higher error on noisy or conversational audio.

This model is Belarusian-monolingual (Belarusian-only tokenizer, trained on Common Voice Belarusian). It will degrade on Russian insertions and on mixed Belarusian-Russian speech; it is not a multilingual model.

Conversion notes

torch 2.10.0 -> torch.jit.trace(strict=False) -> coremltools 9.0 (mlprogram, FLOAT16, macOS15). Three things needed care: the encoder is warmed up with one forward before tracing so the relative positional-encoding cache is built; coremltools 9.0 crashes on the aten::Int pattern under numpy 2 (fixed with a scoped register_torch_op override on int); and dither must be 0 at inference or the random spectrogram perturbation flips borderline emissions.

License and attribution

Licensed under CC-BY-4.0, inherited from the base model nvidia/stt_be_fastconformer_hybrid_large_pc (© NVIDIA, CC-BY-4.0). Changes made: export of the RNN-T path to fp16 Core ML (mlprogram), split into encoder / decoder-step / joint-step packages, with the log-mel front-end reproduced from the checkpoint buffers.

Architecture: FastConformer. Toolkit: NVIDIA NeMo.

Core ML conversion © 2026 Sergey Makarov (smkrv).