granite_speech_4_1_2b

See our collection for all versions of Granite Speech.

Run Granite Speech with Keras 3: JAX, PyTorch, or TensorFlow

zeromodels/granite_speech_4_1_2b

Paper: Granite-speech: open-source speech-aware LLMs with strong English ASR capabilities (arXiv:2505.08699) · HF Papers

Granite Speech is a speech-aware LLM, not ASR with an LM bolted on. A conformer CTC encoder and BLIP-2 style Q-Former turn mel features into audio embeddings that fill <|audio|> placeholders in a Granite decoder. You ask for a transcript, a summary, or an answer in ordinary English; text-only mode keeps the plain Granite decoder.

For more details on the model, please go to the upstream model card.

Pure-Keras 3 conversion of ibm-granite/granite-speech-4.1-2b for zeromodels. One implementation runs unmodified on TensorFlow / Torch / JAX.

This is a speech LLM checkpoint (GraniteSpeechConditionalGenerate, Granite 4.1 2B). Prefer load_dtype="bfloat16".

✨ Quick start

import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

import keras
import numpy as np
import soundfile as sf
from zeromodels.models.granite_speech import (
    GraniteSpeechConditionalGenerate,
    GraniteSpeechProcessor,
)

model = GraniteSpeechConditionalGenerate.from_weights(
    "zeromodels/granite_speech_4_1_2b", load_dtype="bfloat16"
)
processor = GraniteSpeechProcessor.from_weights("zeromodels/granite_speech_4_1_2b")

audio, sr = sf.read("your_audio.wav", dtype="float32")  # 16 kHz mono

# Ask in words: same audio + different instruction => different answer.
conversation = [
    {
        "role": "user",
        "content": [
            {"type": "audio"},
            {
                "type": "text",
                "text": "can you transcribe the speech into a written format?",
            },
        ],
    }
]

inputs = processor(conversation=conversation, audio=audio, sampling_rate=sr)
out = model.generate(**inputs, max_new_tokens=64)
ids = np.asarray(keras.ops.convert_to_numpy(out))[0].tolist()
print(repr(processor.tokenizer.decode(ids)))

Load any Granite Speech variant the same way with from_weights("zeromodels/<variant>"):

Tips

• Set KERAS_BACKEND before importing Keras / zeromodels.
• Pass audio via audio= + sampling_rate=; put only an {"type": "audio"} marker in the conversation.
• Drop audio for text-only chat; the audio LoRA stays off.
• See Granite Speech docs and Loading Weights.
• Community / upstream safetensors still work via the hf: prefix, e.g. GraniteSpeechConditionalGenerate.from_weights("hf:ibm-granite/granite-speech-4.1-2b").

Special Thanks

A huge thank you to the IBM Granite authors for creating and releasing these models.

License: Apache 2.0.