Breeze-ASR-25-mlx-fp16

Breeze-ASR-25 · MLX (fp16)

This repository contains MediaTek-Research/Breeze-ASR-25 converted to MLX, Apple's array framework for Apple Silicon. The source model is a Whisper-large-v2 fine-tune for Taiwanese Mandarin and Mandarin–English code-switching. Its weights are carried over unchanged apart from a cast from bfloat16 to float16, two 16-bit floating-point formats that differ in how they divide bits between exponent and mantissa; there is no fine-tuning, no architectural change, and no integer quantization.

Converting a model does not change what it knows. It changes where and how quickly the model runs, and it can change what the model emits, so this card reports both. On the full ASCEND test set the converted model scores 18.01 % against the 17.74 % its authors report, a difference of 0.27 percentage points. On an Apple M4 Pro it transcribes a five-minute recording in 24 seconds using 4.0 GB of memory, where the source checkpoint running through transformers takes 168 seconds and 28.2 GB.

What the model can do, how it was trained, and its known biases are documented by its authors and are not repeated here. A 4-bit build of the same weights is measured alongside throughout, and §3.3 compares the two.

The repository loads in both MLX Whisper runtimes: mlx-whisper, Apple's reference implementation, and mlx-audio, a community package that servers such as oMLX build on. Most published MLX Whisper repositories work in only one of the two, for reasons given in §2.2.

---

1. Usage

With mlx-audio, and servers built on it

The repository is laid out the way mlx-audio expects, so no preparation is needed:

from mlx_audio.stt.utils import load_model

model = load_model("BRlin/Breeze-ASR-25-mlx-fp16")
print(model.generate("audio.wav", language="zh").text)

With mlx-whisper

mlx-whisper 0.4.3 — the current PyPI release — reads a weight file named weights.safetensors, while servers built on mlx-audio require the name model.safetensors when they scan the local HuggingFace cache. A single file cannot carry both names. This repository ships model.safetensors, following the convention OpenAI's own Whisper repositories use, so users of the released mlx-whisper create an alias once after downloading. A hard link is a second directory entry pointing at the same data, so it costs no additional disk space:

cd <downloaded-directory> && ln model.safetensors weights.safetensors

This step is specific to the released version. mlx-whisper on its main branch already prefers model.safetensors (PR #1399, merged 2025-12-15); verified against a build from that branch, this repository loads with no alias. The change is not in any published release — main still reports version 0.4.3 — so the step remains necessary until one ships.

import mlx_whisper

r = mlx_whisper.transcribe(
    "audio.wav", path_or_hf_repo="<downloaded-directory>", language="zh",
)
print(r["text"])

Repeated runs will not produce identical text

mlx_whisper.transcribe() re-decodes a segment at successively higher temperature when the segment fails its compression-ratio or log-probability checks. Temperature above zero means the decoder samples from the token distribution rather than taking its most probable token, so two runs over the same audio can differ. Across five runs of the ASCEND subset used during development, the spread was 0.13 percentage points.

Passing temperature=0.0 removes the fallback and makes output identical across runs. It also removes the mechanism that limits repetition loops on silence, so the trade differs by task: it is what benchmarking needs, and not what unattended transcription of long recordings needs.

Long or noisy recordings

Whisper models, including this one, emit repeated tokens when they encounter silence under the default thresholds. Tightening those thresholds reduces it:

r = mlx_whisper.transcribe(
    "audio.wav", path_or_hf_repo="<downloaded-directory>", language="zh",
    condition_on_previous_text=False,  # do not carry context across windows
    compression_ratio_threshold=1.8,   # tighter than the 2.4 default
    logprob_threshold=-0.6,            # enter temperature fallback sooner
    no_speech_threshold=0.5,
    hallucination_silence_threshold=2.0,
)

No measurement in §3 used these values; each table there states the decoding settings it was produced under.

What does not work

transformers cannot load this repository. The tensor names follow MLX conventions rather than a PyTorch state dictionary, so the source repository remains the right choice for that path.

---

2. What the conversion did

2.1 Weights

The source checkpoint holds 1,259 tensors in bfloat16 under HuggingFace naming. This build holds 1,258 tensors in float16 under MLX naming. The tensor that is absent is model.encoder.embed_positions.weight: MLX generates the encoder's positional embedding when the module is constructed rather than reading it from the file, so it is not carried over. Every other tensor is present — a load-integrity check reports 1,258 parameters expected against 1,258 tensors in the file, with no missing name, no unexpected name, and no value that is infinite or not-a-number.

Conversion used mlx-audio 0.4.2 on an Apple M4 Pro running macOS 26.2. Two departures from a stock invocation apply to anyone reproducing this build.

The converter's --dtype argument does not reach the model constructor, and the cast it applies afterwards is written to a variable that is discarded before the file is saved. The output is float16 whatever value is supplied; this was observed in mlx-audio 0.4.2 and 0.4.5. float16 is the intended target here in any case, but the argument cannot be relied on to select a different one.

The config.json that the converter writes is a verbatim copy of the source repository's HuggingFace configuration. mlx-whisper parses config.json into a ten-field structure and raises TypeError on any field outside it, so the file was rewritten into that schema. mlx-audio reads either schema, so this change costs nothing there.

The 4-bit build referenced throughout was produced from the same source with affine quantization at 4 bits and a group size of 64, meaning each group of 64 weights shares one scale and offset.

2.2 Why float16

mlx-whisper accepts float16 or float32 and rejects bfloat16: bfloat16 weights multiplied against a float16 input promote to float32, which fails an assertion the decoder makes on the encoder's output dtype. mlx-audio constructs its Whisper module in float16 by default. float16 is therefore the format both runtimes read without modification, and is what OpenAI and mlx-community publish for Whisper.

A float32 build was produced and measured. It scores within run-to-run variation of this one at twice the file size, and cannot be loaded through mlx-whisper's default call path, which assumes float16 unless told otherwise. It is not published.

---

3. Measurements

3.1 Evaluation data

Accuracy is measured on two public corpora. Both appear in the upstream card's results tables, so the figures below can be placed beside the ones its authors publish.

CAiRE/ASCEND is a corpus of spontaneous Mandarin–English conversation with expert transcripts, released under CC BY-SA 4.0. The complete test split is used: 1,315 utterances totalling 55.0 minutes, with a median utterance of 1.9 seconds and none longer than 11.8 seconds. Each utterance carries a language label — 685 zh, 257 en, 373 mixed — and results are broken out by that label because the model behaves differently across them. Its reference transcripts are written in Simplified Chinese.

ky552/ML2021_ASR_ST holds recordings from National Taiwan University's 2021 Machine Learning course: Mandarin lecturing carrying English technical vocabulary, a register closer to meeting or lecture transcription than ASCEND's conversation. The upstream card reports it under the name ML-lecture-2021-long. The released test split is 14,916 short clips, each tagged with the lecture it came from; concatenating one lecture's clips in released order reconstructs continuous audio. Lecture week7_1 is used here, 1,676 clips totalling 57.4 minutes, with its first 300 seconds serving as the throughput excerpt. Its reference transcripts are written in Traditional Chinese. The dataset card declares no licence; the upstream table lists the corpus as MIT.

3.2 How error rates were computed

The upstream card reports word error rate, the edit distance between the transcript and the reference divided by the number of reference words. Counting words in Mandarin–English text requires deciding where word boundaries fall, and the decision made upstream is not published. The figures here therefore use a mixed error rate: each Chinese character counts as one unit, each unbroken run of Latin letters or digits counts as one unit, and the edit distance over that sequence is divided by the number of reference units. Punctuation is removed, case is folded, and [UNK] markers — annotations for speech the transcriber could not resolve, not words the model should produce — are deleted from the reference.

This model writes Traditional Chinese. ASCEND's references are Simplified and NTUML2021's are Traditional, so hypotheses are converted to Simplified before scoring against ASCEND and left unconverted for NTUML2021. The size of that decision is worth stating: scoring ASCEND without the conversion reports its zh subset at 42.33 % rather than 16.85 %, and applying the conversion to NTUML2021 reports 27.72 % rather than 5.91 %. Both wrong figures were produced during this evaluation by carrying one corpus's setting to the other.

3.3 Accuracy

The table below places this build beside the figures the upstream authors publish. Both columns are error rates, so lower is better; Δ is their difference in percentage points, positive meaning this build scores worse. The upstream column is word error rate as published; ours is the mixed error rate described above, and §3.5 discusses what that difference in definition permits.

Overall agreement is within 0.27 pp. The en subset shows the widest difference and rests on the least data — 257 utterances, 1,279 reference units — so it carries correspondingly less weight, and whether it reflects model behaviour or a difference in how words were counted is not something this evaluation can separate.

On long-form audio, the same measurement over the reconstructed 57.4-minute lecture gives 5.91 % for this build and 5.82 % for the 4-bit build, against the 4.98 % upstream reports for ML-lecture-2021-long. Upstream does not state which lectures that figure covers and this measurement uses one of fourteen, so part of the difference is a difference in material rather than in the model, and this comparison is weaker than the ASCEND one.

3.4 Throughput and memory

Speed is reported as real-time factor: processing time divided by audio duration, so 0.08 means a minute of audio takes about five seconds and lower is faster. Peak memory is the highest allocation observed during the run.

Throughput on this machine depends on what ran before it — a configuration measured second on a warm device runs roughly 1.3× slower than the same configuration measured first — so the four configurations were run in order and then immediately in reverse, giving each the same average position. Each figure below is the median of four timed runs on a 300-second excerpt of the public lecture recording, with a warm-up run discarded. The range column shows the fastest and slowest of those four.

Against the source checkpoint, this build processes the excerpt 7.1× faster in 7.1× less memory, and the 4-bit build 10.0× faster in 15.7× less memory. The two MLX runtimes differ by 2.6× on identical weights: mlx-audio serves many model families through one interface, while mlx-whisper implements Whisper alone.

The source row runs in float32 because transformers with float16 on MPS — PyTorch's Metal backend — produces output that is not usable for this architecture, emitting fragments such as '!' and '<|fr|>!' in place of speech. float32 is therefore the only working configuration on that path, and its memory figure reflects that rather than a property of the model.

These figures describe a five-minute recording. Over the full 57.4-minute reconstruction the same fp16 build measured RTF 0.102 at 5.63 GB peak, so both cost and memory were higher on the longer material; two lengths are not enough to describe how the relationship behaves in between.

3.5 Choosing between fp16 and 4-bit

Both builds hold the same weights; the 4-bit build stores them at lower precision. The accuracy rows below use temperature=0.0 on both, which makes each figure exact rather than an average over runs that differ.

The two builds separate by 1.29 pp on ASCEND's short conversational utterances and by 0.09 pp on the lecture recording, in opposite directions. A single lecture is not enough to establish that they are equivalent on long-form material, but the cost of quantization clearly is not constant across the two kinds of audio, and a single number would not describe it.

The last row counts a different kind of failure. Error rate charges substitutions, insertions and deletions alike, so an utterance the model declines to transcribe and one it transcribes with several words wrong can contribute similarly, though they are not equivalent in use. The 4-bit build returns nothing about twice as often, and ASCEND's very short utterances — its median is 1.9 seconds — are the material most affected.

This repository holds the fp16 build; the 4-bit build is published separately at BRlin/Breeze-ASR-25-mlx-4bit and is reproducible from the parameters in §2.1.

---

4. Limitations

Coverage. Two corpora, both Mandarin–English. Nothing here measures other languages, other speech registers, or the Formosa long-form corpora the upstream card also reports. The long-form figure rests on one lecture of fourteen.

Metric comparability. The word segmentation behind the published upstream figures is not available, so the comparison in §3.3 places two similarly-defined but not identical metrics side by side. The 0.27 pp overall agreement is consistent with the definitions being close; it does not demonstrate that they are the same, and per-subset differences of one to two points may originate there.

Runtime coverage. Accuracy was measured through mlx-whisper only. The mlx-audio row in §3.4 reports throughput without a corresponding error rate.

Version coverage. Behaviour was verified against mlx-audio 0.4.2 and 0.4.5. Version 0.4.6 and later were not tested.

Not measured. Word-level timestamps, including the alignment quality the upstream card describes; streaming or real-time operation; accuracy under the tightened thresholds shown in §1.

Limitations that belong to the base model rather than to this conversion are documented upstream.

---

5. Licence and attribution

Apache-2.0, inherited from the source model. This build adds no restrictions.

The model is the work of MediaTek Research. This repository contributes the MLX build and the measurements above.

• Model: MediaTek-Research/Breeze-ASR-25 · paper · code
• Base architecture: openai/whisper-large-v2
• Evaluation data: CAiRE/ASCEND · ky552/ML2021_ASR_ST
• Runtimes: ml-explore/mlx-examples · Blaizzy/mlx-audio

@article{breeze-asr-25,
  title   = {A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Data},
  journal = {arXiv preprint arXiv:2506.11130},
  year    = {2025}
}