moonshine-tiny-en-onnx

Moonshine Tiny (English) — ONNX export

ONNX graphs for moonshine-ai/moonshine-tiny, exported for on-device speech recognition through ONNX Runtime. The model, its training data and its design belong to Moonshine AI; this repository only converts what they published, under their MIT licence.

The mirror of this for Ukrainian is vasyadeva/moonshine-tiny-uk-onnx — same graph shapes, same input and output names, so an application can switch language by swapping the files and nothing else. Note the licences differ: the Ukrainian model is under the Moonshine AI Community License, this one is MIT.

Source

• Model: moonshine-ai/moonshine-tiny — 27.1M parameters, English
• Authors: Moonshine AI (formerly Useful Sensors)
• Paper: arxiv 2410.15608

Please cite and support the original work.

Why ONNX, and why this shape

Moonshine takes raw 16 kHz audio and does not pad it to a fixed window, so a two-second utterance costs two seconds of work. That is the reason to prefer it on a phone over a Whisper-family model, whose encoder always processes its 30-second window and charges the same for a short command as for a long sentence — measured in a real app, Whisper small needed 45.9 s for a 3.3 s question on an iPhone 14 Pro Max where this shape needs a fraction of a second.

Three graphs, so the decoder's attention cache survives between tokens:

Weights live in the .data sidecars; ONNX Runtime loads them from beside the graph, so keep each pair together.

No int8 set here, deliberately. Quantizing everything turns the encoder's convolutions into ConvInteger, which the ONNX Runtime builds that ship inside mobile apps have no kernel for — the session refuses to open. Quantizing only the matrix multiplies works and is faster, but the embedding table stays float either way, so the saving is smaller than it looks, and on the Ukrainian model it cost a word of accuracy on one of three samples.

Inputs / outputs

encoder.onnx
  input_values          float32 [batch, samples]        raw waveform, 16 kHz, mono
  → encoder_hidden_states float32 [batch, frames, 288]  ~41 frames per second

decoder_init.onnx
  input_ids             int64  [batch, 1]               decoder_start_token_id (1)
  encoder_hidden_states float32 [batch, frames, 288]
  → logits              float32 [batch, 1, 32768]
  → present_{key,value}_self_{0..5}    [batch, 8, 1, 36]
  → present_{key,value}_cross_{0..5}   [batch, 8, frames, 36]

decoder_step.onnx
  input_ids             int64  [batch, 1]               previous token
  cache_position        int64  [1]                      absolute index of this token
  encoder_hidden_states float32 [batch, frames, 288]
  past_{key,value}_self_{0..5}, past_{key,value}_cross_{0..5}
  → logits, present_{key,value}_self_{0..5}

Greedy decoding: run the encoder once, decoder_init once, then decoder_step until the argmax is EOS (2). Feed the self-attention KV back each step; the cross-attention KV does not change within an utterance.

Three things that will cost you an evening if nobody says them

1. decoder_step needs encoder_hidden_states even though the cross-attention KV is already cached. Without it the decoder skips cross-attention and produces fluent, confident, entirely invented text. It does not error.
2. cache_position is the absolute index of the current token — 1 for the first generated one, not 0.
3. The export needs attn_implementation="eager"; with the default SDPA path torch.onnx.export fails on enable_gqa=True where query and key heads are equal.

What was verified

Against the PyTorch model, on this exact export: three Fleurs en_us utterances decoded identically to model.generate(), and the encoder took 0.14–0.26 s for 9–11 seconds of audio on an x86 CPU in Docker.

Reproducing

export_encoder.py, export_cached_param.py and validate_vs_torch.py are included; set MS_MODEL and MS_OUT. Needs torch 2.8, transformers 5.0, onnx, onnxscript, onnxruntime.

License

MIT, as the source model is. The rights to the model are Moonshine AI's.