ainotes-whisper-bengali-medium-q5_1

Whisper Bengali medium, GGML q5_1

bengaliAI/tugstugi_bengaliai-asr_whisper-medium converted to GGML and quantised to q5_1 for whisper.cpp, for use on Android.

An optional higher-accuracy alternative to the 190 MB small model, not a replacement for it.

Measured

FLEURS bn_in, 128 clips, both models scored in one run through whisper.cpp with the settings the app actually uses - no_timestamps, greedy decoding, common-script romanisation before scoring. The small model measured here is the q5_1 file the app ships, not a reconversion.

Both figures had to improve for this to be published. That rule exists because a Hinglish fine-tune once won on one axis while scoring three times worse on the other, and was nearly shipped on the strength of the half it won.

Cost

About 200 seconds per note on a Realme RMX2170 (2 big cores at 2.3 GHz), against 33 for the small model - roughly six times the wait. Whisper pads every input to a 30-second window, so that is per note rather than per second of speech: a four-second note costs the same as a twenty-nine-second one.

Measured, not estimated: five runs on device at the settings above gave 193, 193, 201, 220 and 221 seconds. An earlier figure of 89 seconds appeared in these cards and does not reproduce. It is not memory pressure - available RAM held at 2.7 GB throughout. A medium is about 3.6x the arithmetic of a small model and costs 6x the time, which points at memory bandwidth on a phone.

The same figure applies to every language here: one architecture at one size, all exactly 586,572,019 bytes.

A benchmark that disagrees

Scored on three corpora, the shipping small model and this one:

The third row is why this model was briefly rejected. Measured only against it, the medium looks a point worse and the gain looks like benchmark contamination. But the anomaly there is the SMALL model's 9.2%, against 32-43% on two unrelated corpora - that dataset appears to overlap its training data, which flattered the baseline rather than the candidate.

Worth stating because the same trap catches anyone testing an Indic model: one control corpus is not a control. It took a third to show which of two disagreeing benchmarks was the strange one.

Settings

no_timestamps is required. These fine-tunes are trained with timestamp prediction disabled, so their timestamp tokens are untrained and whisper.cpp derails when it trusts them - 47.5% WER against 14.9% on the small Hindi model, with the decoder abandoning the audio for fluent unrelated text.

Attribution

Base model Apache-2.0, credit to its authors. Converted with whisper.cpp's convert-h5-to-ggml.py; token table verified byte for byte against ggerganov's ggml-small.