cohere-transcribe-03-2026-mlx-4bit

Cohere Transcribe 03-2026 — MLX 4-bit

A community MLX 4-bit conversion of CohereLabs/cohere-transcribe-03-2026 for local speech recognition on Apple Silicon, using mlx-audio 0.5.5. Converted directly from the official BF16 checkpoint; no intermediate quantized checkpoint and no fine-tuning are used.

The weight file is 1.507 GB (1.404 GiB). Quantization uses 4-bit affine weights with group size 64 for supported layers; unquantized tensors remain BF16. This is not an all-tensors-4-bit model. The main benefit is reduced storage and memory use.

Usage

Requires an Apple Silicon Mac, Python 3.12, and FFmpeg on PATH. Download this repository into a folder named cohere-transcribe-03-2026-mlx-4bit. Run the commands and Python example below from its parent directory. The verified runtime is:

python3.12 -m venv .venv
source .venv/bin/activate
pip install -r ./cohere-transcribe-03-2026-mlx-4bit/requirements.txt
brew install ffmpeg

Set model_dir to the downloaded folder and audio_path to your recording:

from pathlib import Path
import subprocess

import numpy as np
from mlx_audio.stt.utils import load_model

model_dir = Path("./cohere-transcribe-03-2026-mlx-4bit")
audio_path = Path("./input.wav")

# Decode/resample to 16 kHz mono; no enhancement is applied.
decoded = subprocess.run(
    ["ffmpeg", "-v", "error", "-i", str(audio_path), "-vn",
     "-ac", "1", "-ar", "16000", "-f", "f32le", "pipe:1"],
    check=True, capture_output=True,
)
audio = np.frombuffer(decoded.stdout, dtype="<f4").copy()
model = load_model(model_dir, strict=True)
result = model.generate(
    audio, sample_rate=16000, language="en", punctuation=True,
    batch_size=4, max_tokens=256,
)
print(result.text)

After installing dependencies, you can also replace the local loading line with the public repository ID to download automatically:

model = load_model("lyzgeorge/cohere-transcribe-03-2026-mlx-4bit", strict=True)

Once the model and dependencies are downloaded, inference works offline. This checkpoint is intended for mlx_audio.stt.utils.load_model; do not load the quantized weights with Transformers AutoModel. Upstream Transformers source files are retained for provenance, but this usage path runs mlx-audio's native implementation.

Specify the spoken language explicitly. Supported language codes inherited from upstream: en, fr, de, es, it, pt, nl, pl, el, ar, ja, zh, vi, ko. Only English was evaluated here. This workflow does not provide automatic language detection, translation, speaker diarization, or word-level timestamps.

Long recordings are split at low-energy points into clips of at most 35 seconds. The example uses greedy decoding, punctuation enabled, batch size 4, and a maximum of 256 generated tokens per clip. Use batch size 1 to reduce memory use. Returned segment boundaries describe audio chunks, not precise word alignment.

Local evaluation

Measured on Apple M4, 24 GB unified memory, macOS Darwin 24.6.0, with mlx-audio==0.5.5 and mlx==0.32.2, on one 600-second accented-English lecture. Both checkpoints used the same audio, 16 kHz mono preprocessing, 35-second maximum clips, batch size 4, and a 256-token cap.

The reference transcript was confirmed as human-checked by the project owner. WER measures differences from that reference: remove Speaker N: labels, lowercase, retain tokens matching [a-z0-9']+, then compute word-level edit distance divided by 1,297 reference words. The 0.23 percentage-point gap corresponds to three additional edit operations overall; it does not mean that the two outputs differ in only three words.

Timing covers one complete model.generate() call after a five-second audio warm-up, with the model already loaded and audio already decoded. It excludes download, model loading, and FFmpeg preprocessing. Peak memory is measured by MLX after resetting its peak counter before generation; it includes model allocations but is not whole-process RSS or total system memory. GB means 10^9 bytes; GiB means 2^30 bytes.

These are single-recording, single-run observations, not a standardized benchmark or a statistically established accuracy comparison. They indicate approximately 63% smaller weights and 50% lower MLX peak memory here, with about 6% lower inference time. They do not establish lossless quantization, superiority over other community conversions, or performance across all supported languages. Batch size 1 measured 1.91 GiB and 18.12 seconds on the same recording.

Release validation

Validated on 2026-09-24 in a newly created Python 3.12.11 virtual environment with the pinned dependencies in requirements.txt, using a separate copy of the model folder, an empty Hugging Face cache, offline flags, and blocked Python socket connections.

• The Python example above executed successfully with strict weight loading on the bundled upstream demo audio.
• The full 600-second English recording produced 19 nonempty segments covering the complete recording, with text identical to the earlier 4-bit evaluation output.
• Weight-index keys matched the safetensors file; model-card languages matched the model configuration.
• No Hugging Face cache files were created. The public repository was subsequently downloaded anonymously into a separate cache; the downloaded weight SHA-256 matched, strict loading succeeded, and the demo transcript matched the local validation. See published-verification.json for the tested artifact revision.

See validation.json for the runtime versions, checks, and weight-file SHA-256. This was a functional check, not a new performance benchmark; another transcription job was present on the machine.

Conversion and provenance

• Original model: approximately 2 billion parameters, by Cohere Labs.
• Source: CohereLabs/cohere-transcribe-03-2026.
• Exact source revision: b1eacc2686a3d08ceaae5f24a88b1d519620bc09.
• Converter: mlx_audio.convert, from mlx-audio==0.5.5.
• Quantization: affine, 4 bits, group size 64.
• Unquantized tensors: bfloat16, preserving the source precision for those tensors.
• Machine-readable provenance: conversion.json.

To reproduce the conversion, obtain the original snapshot at the revision above (the upstream repository currently requires Hugging Face access), then run:

mlx_audio.convert \
  --hf-path /path/to/official/snapshot \
  --mlx-path ./cohere-transcribe-03-2026-mlx-4bit \
  --model-domain stt \
  -q --q-bits 4 --q-group-size 64 --q-mode affine \
  --dtype bfloat16

Limitations

Recognition can contain omissions, substitutions, and hallucinated text, particularly on unfamiliar domains, noise, or silence. Review transcripts where accuracy matters. No new multilingual, noisy-audio, or silence evaluation was conducted for this release. Streaming generation is not implemented in this runtime's Cohere backend. The optional VAD path failed in the local mlx-audio 0.5.5 experiment; the example uses the default energy-based segmentation instead.

License and attribution

Apache License 2.0, aligned with the original model's declared license. See LICENSE, ATTRIBUTION.md, and the upstream model repository.

Original model and accompanying upstream source files: Cohere Labs and their respective contributors. This community artifact changes the weight representation to MLX affine 4-bit, adds quantization metadata, and supplies conversion provenance and usage/evaluation documentation. It is not an official Cohere release or endorsement. Original source files and existing attribution notices are retained.