granite_speech_4_1_2b_plus

See our collection for all versions of Granite Speech Plus.

Run Granite Speech Plus with Keras 3: JAX, PyTorch, or TensorFlow

zeromodels/granite_speech_4_1_2b_plus

Paper: Granite-speech: open-source speech-aware LLMs with strong English ASR capabilities (arXiv:2505.08699) · HF Papers

Granite Speech Plus is the Granite 4.0-based speech-aware LLM successor to Granite Speech: a conformer CTC encoder and Q-Former projector feed audio embeddings into <|audio|> slots of a Granite decoder. You ask for what you want in words (transcribe, summarize, answer). LoRA is fully merged; no adapter toggle.

For more details on the model, please go to the upstream model card.

Pure-Keras 3 conversion of ibm-granite/granite-speech-4.1-2b-plus for zeromodels. One implementation runs unmodified on TensorFlow / Torch / JAX.

This is a speech LLM checkpoint (GraniteSpeechPlusConditionalGenerate) on Granite 4.0 2B. Prefer load_dtype="bfloat16".

✨ Quick start

import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

import keras
import numpy as np
import soundfile as sf
from zeromodels.models.granite_speech_plus import (
    GraniteSpeechPlusConditionalGenerate,
    GraniteSpeechPlusProcessor,
)

model = GraniteSpeechPlusConditionalGenerate.from_weights(
    "zeromodels/granite_speech_4_1_2b_plus", load_dtype="bfloat16"
)
processor = GraniteSpeechPlusProcessor.from_weights("zeromodels/granite_speech_4_1_2b_plus")

audio, sr = sf.read("your_audio.wav", dtype="float32")  # 16 kHz mono

# Instruction in words: this is a speech LLM, not fixed-task ASR.
conversation = [
    {
        "role": "user",
        "content": [
            {"type": "audio"},
            {
                "type": "text",
                "text": "can you transcribe the speech into a written format?",
            },
        ],
    }
]

inputs = processor(conversation=conversation, audio=audio, sampling_rate=sr)
out = model.generate(**inputs, max_new_tokens=64)
ids = np.asarray(keras.ops.convert_to_numpy(out))[0].tolist()
print(repr(processor.tokenizer.decode(ids)))

Load any Granite Speech Plus variant the same way with from_weights("zeromodels/<variant>"):

Tips

• Set KERAS_BACKEND before importing Keras / zeromodels.
• Pass audio via audio= + sampling_rate=; put only an {"type": "audio"} marker in the conversation (do not embed the waveform).
• Change the text instruction to get a different answer over the same clip.
• See Granite Speech Plus docs and Loading Weights.
• Community / upstream safetensors still work via the hf: prefix, e.g. GraniteSpeechPlusConditionalGenerate.from_weights("hf:ibm-granite/granite-speech-4.1-2b-plus").

Special Thanks

A huge thank you to the IBM Granite authors for creating and releasing these models.

License: Apache 2.0.