LFG-2

LFG-2

LFG-2 is a speech-in / text-out voice assistant: a trained audio projector bridging a Gemma E4B audio encoder to a Gemma 4 31B language model. The audio encoder and the language model are frozen; only the projector is trained, so the model inherits the full text-side reasoning of Gemma 4 31B while learning to listen.

• Input: spoken English audio (16 kHz mono).
• Output: text. The model reasons in a hidden thought channel (emitting a <heard>…</heard> transcription of what it heard as a comprehension check), then produces the final answer.

Usage

pip install -U transformers accelerate librosa

# On a bare machine, install the torch trio together first (matching CUDA build):
#   pip install torch torchvision torchaudio

LFG-2 loads exactly like base Gemma 4 E4B audio — AutoProcessor + AutoModelForMultimodalLM — the only difference is trust_remote_code=True, which lets the repo's bundled model class install the trained projector for you.

import torch
from transformers import AutoProcessor, AutoModelForMultimodalLM

REPO = "glenn2/LFG-2"
processor = AutoProcessor.from_pretrained(REPO)
model = AutoModelForMultimodalLM.from_pretrained(
    REPO, dtype="auto", device_map="auto", trust_remote_code=True,
)

# Ask a question with your voice (audio can be a local path or a URL; 16 kHz mono).
messages = [
    {"role": "system", "content": "You are a helpful voice assistant. Answer the user's spoken question clearly and concisely."},
    {"role": "user", "content": [{"type": "audio", "audio": "question.wav"}]},
]
inputs = processor.apply_chat_template(
    messages, tokenize=True, return_dict=True, return_tensors="pt",
    add_generation_prompt=True, enable_thinking=True,
).to(model.device)
input_len = inputs["input_ids"].shape[-1]

outputs = model.generate(**inputs, max_new_tokens=2048)
response = processor.decode(outputs[0][input_len:], skip_special_tokens=False)

# parse_response strips the <|channel>thought…<channel|> block; keep the answer.
print(processor.parse_response(response)["content"])

<details> <summary>Without <code>trust_remote_code</code> (manual projector install)</summary>

If you'd rather not run repo code, load the base model normally and install the projector yourself (needs huggingface_hub):

import sys, torch
from huggingface_hub import snapshot_download
from transformers import AutoProcessor, AutoModelForMultimodalLM

local = snapshot_download("glenn2/LFG-2")
processor = AutoProcessor.from_pretrained(local)
model = AutoModelForMultimodalLM.from_pretrained(local, dtype="auto", device_map="auto")

sys.path.insert(0, local)
from deep_projector import install_deep_projector
ckpt = torch.load(f"{local}/projector_final.pt", map_location=model.device)
deep = install_deep_projector(model, hidden=ckpt["config"]["hidden"],
                              n_hidden_layers=ckpt["config"]["mlp_layers"])
deep.load_state_dict(ckpt["state_dict"], strict=True)
# ... then apply_chat_template / generate / parse_response as above.

</details>

Notes

• Audio format: 16 kHz mono. Resample first (e.g. librosa.load(path, sr=16000)).
• Thinking: with enable_thinking=True the model emits <|channel>thought … <channel|> (including a <heard>…</heard> transcript of the audio) before the answer. processor.parse_response(...)["content"] returns just the final answer; strip any residual <heard>…</heard> if present.
• Decoding: greedy (do_sample=False) is reproducible and used for all benchmark numbers below. For more varied generation use Gemma-4 sampling: do_sample=True, temperature=1.0, top_p=0.95, top_k=64.
• Stop token: generation ends on <turn|> (the Gemma 4 turn terminator).
• Long answers / runaway thinking: cap total length with max_new_tokens and optionally force-close the thought channel after a fixed budget by writing a small LogitsProcessor that forces the <channel|> token once N tokens have been generated (see deep_projector.py / the training repo for the reference ThinkingBudgetProcessor).

VoiceBench

Evaluated with the official VoiceBench protocol (greedy decoding, thinking enabled).

Intended use & limitations

• Designed for English spoken questions/instructions → text answers.
• Not a transcription service (though it transcribes internally); not intended for non-speech audio, speaker ID, or languages other than English.