TinyMOSS-Diarize

TinyMOSS-Diarize

MOSS-Transcribe-Diarize 0.9B, quantized to an average of 2.911 bits per value — 1.8 GB down to 316 MiB — and still doing all three jobs in a single pass: transcription, timestamps, and speaker labels, in nine languages.

The weights here are stored packed. With the Arm NEON kernels in the companion repo they are multiplied as packed data, so a 4 GB Raspberry Pi can run a model it otherwise cannot load.

Quick start

pip install torch transformers soundfile
python run_inference.py your_audio.wav --device cpu

Output looks like this — timestamps and speaker tags come from the model itself, not a separate diarization pass:

[0.48][S01] He hoped there would be stew for dinner, turnips and carrots and bruised
potatoes and fat mutton pieces to be ladled out in thick, peppered flour fat and sauce.[10.34]

SHA256SUMS covers every file in this repo.

How the weights are stored

Unpacking is bit-identical to the reference implementation (max_abs = 0), and the loader verifies all 341 packed tensors by checksum before use.

Accuracy

16 datasets, 553 utterances, zero inference errors and zero malformed transcripts under the grammar-constrained decoder.

Multi-speaker audio is where this model is worth its size. Whisper transcribes only, so it is scored on text alone and still trails badly:

Single speaker, quantization costs us real accuracy:

FLEURS (30 utterances per language): en 9.64%, de 16.00%, es 7.10%, fr 18.42%, it 10.87%, ko 23.57%, ru 19.40% WER; ja 12.10%, zh 11.12% CER. Whisper-turbo is better than this model on every FLEURS language — the trade here is size and diarization, not raw monolingual accuracy.

On-device performance

Same model, same audio, same machine; the only difference is whether weights stay packed.

Apple M4 MacBook Air, 10.4 s clip:

Raspberry Pi 5, 4 GB, 5.2 s clip: packed finishes in 142.7 s (load 8.9 s, peak 3,077 MiB). The same model dequantized to BF16 never finished — 177 s just to load, still on its first token when killed at 628 s. FP32 managed one token before timing out.

Every on-device transcript matched the GPU reference character for character. Benchmarks are 1 warmup

• 3 timed runs, median. Raw data and the kernels are in the GitHub repo.

Limitations

• "2.911 bits" is an average across the package, not a uniform 3-bit model.
• Meeting-room diarization is the hardest regime; AliMeeting timestamp error averages 6.22 s.
• The decoder guarantees a parseable transcript; it does not infer or repair speaker identity.
• Clean single-speaker accuracy is measurably worse than the BF16 original.

License and attribution

Apache-2.0. Built on OpenMOSS/MOSS-Transcribe-Diarize (Apache-2.0). Quantization, packed-weight runtime, and Arm kernels added on top.