granite-speech-4.1-2b-mlx-6bit

Granite Speech 4.1 2B MLX 6-bit

Quality-oriented MLX conversion of ibm-granite/granite-speech-4.1-2b for Apple Silicon. This is the autoregressive Granite Speech model, not the separate NAR variant.

The model runs with mlx-audio and supports multilingual automatic speech recognition (ASR), speech translation (AST), punctuation and capitalization, and keyword-biased transcription.

Quantization

This conversion uses post-training weight quantization only. No training, fine-tuning, calibration dataset, or importance matrix was used.

The acoustic stack is kept in BF16 because quantization errors can compound through the Conformer and alter the embeddings supplied to the language model. The token embedding and output head are also kept in BF16 to preserve input and output token precision. This follows the general multimodal quantization practice of protecting encoders, projectors, embeddings, and output tensors while compressing the large internal language-model projections.

• Source revision: de575db64086f84fdc79da4932d1076e965bc546
• Effective average reported by MLX: 10.144 bits per weight
• model.safetensors: approximately 2.7 GB

Requirements

• Apple Silicon Mac
• macOS 14 or later
• mlx-audio >= 0.4.5

pip install -U mlx-audio

Usage

python -m mlx_audio.stt.generate \
  --model divydeep/granite-speech-4.1-2b-mlx-6bit \
  --audio audio.wav \
  --output-path transcript \
  --format txt \
  --prompt "transcribe the speech with proper punctuation and capitalization."

Python:

from mlx_audio.stt.utils import load_model

model = load_model("divydeep/granite-speech-4.1-2b-mlx-6bit")
result = model.generate(
    "audio.wav",
    prompt="transcribe the speech with proper punctuation and capitalization.",
    temperature=0.0,
)
print(result.text)

Greedy decoding with temperature=0.0 is recommended for transcription.

Prompts

For non-English punctuation, translation, and keyword biasing, use the English prompt forms recommended by IBM.

Validation

The conversion was validated by:

• Strictly loading all weights with mlx-audio.
• Confirming the architecture is granite_speech, not granite_speech_nar.
• Running greedy transcription on IBM's bundled multilingual_sample.wav.
• Checking preservation of English and French text, punctuation, accents, and capitalization in the generated transcript.

This is a conversion smoke test, not a full WER benchmark. The upstream IBM evaluation results describe the original model and should not be interpreted as measured results for this quantized conversion.

Limitations

• Quantization may affect rare words, names, punctuation, casing, keyword biasing, translation, and difficult or noisy audio.
• This model has not been independently evaluated on the complete IBM benchmark suite.
• The model is autoregressive and does not provide the throughput behavior of the separate NAR architecture.
• Refer to the upstream model card for intended use, language coverage, safety considerations, training data, and architectural details.

References

• Original model and authoritative model card
• Granite Speech paper
• MLX Audio

License

Apache-2.0, matching the original model.

Citation

@misc{granite-speech-4.1-2b,
  title={Granite 4.1 Speech},
  author={IBM Granite Speech Team},
  year={2026},
  url={https://huggingface.co/ibm-granite/granite-speech-4.1-2b}
}