granite-speech-4.1-2b-mlx-5bit

Granite Speech 4.1 2B MLX 5-bit

Quality-oriented mixed-precision MLX conversion of ibm-granite/granite-speech-4.1-2b for Apple Silicon. This is the autoregressive Granite Speech model, not the separate NAR variant.

The model runs with mlx-audio and supports multilingual automatic speech recognition (ASR), speech translation (AST), punctuation and capitalization, and keyword-biased transcription.

Quantization

This conversion uses post-training weight quantization only. No training, fine-tuning, calibration dataset, or importance matrix was used.

Value and down projections receive extra precision because mixed-tensor quantizers commonly protect these sensitive paths. The acoustic stack, token embedding, and output head remain BF16. This is conceptually similar to medium mixed-tensor quantizations such as GGUF Q5_K_M, but the file format and numerical representation are MLX affine quantization rather than GGUF K-quants.

• Source revision: de575db64086f84fdc79da4932d1076e965bc546
• Effective average reported by MLX: 9.691 bits per weight
• model.safetensors: approximately 2.5 GB

Requirements

• Apple Silicon Mac
• macOS 14 or later
• mlx-audio >= 0.4.5

pip install -U mlx-audio

Usage

python -m mlx_audio.stt.generate \
  --model divydeep/granite-speech-4.1-2b-mlx-5bit \
  --audio audio.wav \
  --output-path transcript \
  --format txt \
  --prompt "transcribe the speech with proper punctuation and capitalization."

Python:

from mlx_audio.stt.utils import load_model

model = load_model("divydeep/granite-speech-4.1-2b-mlx-5bit")
result = model.generate(
    "audio.wav",
    prompt="transcribe the speech with proper punctuation and capitalization.",
    temperature=0.0,
)
print(result.text)

Greedy decoding with temperature=0.0 is recommended for transcription.

Prompts

For non-English punctuation, translation, and keyword biasing, use the English prompt forms recommended by IBM.

Validation

The conversion was validated by:

• Strictly loading all weights with mlx-audio.
• Confirming the architecture is granite_speech, not granite_speech_nar.
• Running greedy transcription on IBM's bundled multilingual_sample.wav.
• Checking preservation of English and French text, punctuation, accents, and capitalization in the generated transcript.

This is a conversion smoke test, not a full WER benchmark. The upstream IBM evaluation results describe the original model and should not be interpreted as measured results for this quantized conversion.

Limitations

• Quantization may affect rare words, names, punctuation, casing, keyword biasing, translation, and difficult or noisy audio.
• This model has not been independently evaluated on the complete IBM benchmark suite.
• The model is autoregressive and does not provide the throughput behavior of the separate NAR architecture.
• Refer to the upstream model card for intended use, language coverage, safety considerations, training data, and architectural details.

References

• Original model and authoritative model card
• Granite Speech paper
• MLX Audio

License

Apache-2.0, matching the original model.

Citation

@misc{granite-speech-4.1-2b,
  title={Granite 4.1 Speech},
  author={IBM Granite Speech Team},
  year={2026},
  url={https://huggingface.co/ibm-granite/granite-speech-4.1-2b}
}