gemma-4-E2B-duplex

Gemma-4-E2B-Duplex

A frame-synchronous full-duplex speech LoRA for google/gemma-4-E2B-it. It listens and "thinks" on the same clock as the incoming audio, so turn-taking, barge-in (interrupting the model mid-reply), and multi-turn conversation are handled natively by the model — not by an external VAD or scripting layer.

The model emits text frame-by-frame; speech is produced by any downstream TTS. This keeps the adapter tiny (24M trainable params, ~96MB) while the base model does the heavy lifting.

Portfolio / technical-sharing release. Runnable demo web app: companion repo https://github.com/allen2c/gemma-4-E2B-duplex.

What it does

• Turn-taking — stays silent while you speak, opens up when you finish
• Barge-in — stop mid-reply the moment you interrupt
• Multi-turn — holds a conversation across many turns without going silent
• Tool calling — emits structured tool calls and speaks the result
• Text input — accepts typed messages mid-conversation

All in one adapter, bilingual (Chinese / English).

Frame-synchronous: the core difference from base Gemma

Base Gemma is turn-based: you send a whole message, it generates a whole reply, done. There is no notion of time — nothing happens "while you are still talking."

This adapter makes the same model frame-synchronous. Time is sliced into 80ms frames, and on every single frame the model does two things at once: it reads the latest audio and decides what to emit. Each frame is 2 audio tokens + 1 text token, and that one text token is a live decision:

time  →  frame 1     frame 2     frame 3     …     frame k     frame k+1   …
audio    [you  ......  speaking  ..............  pause]        (silence)
model    <wait>      <wait>      <wait>      …   <wait>        "Sure,"     "here's"…
              └─ stays quiet while you talk ─┘        └─ opens up on its own ─┘

• Emitting <wait> = "keep listening." Emitting a word = "I'm speaking now."
• Turn-taking is just: when do I switch from <wait> to words?
• Barge-in is: your voice returns mid-reply → the model flips back to <wait> within a few frames.
• Multi-turn is: keep doing this correctly as the conversation state evolves.

Because every decision lives on a shared clock with the audio, these behaviors are the model's own, learned end-to-end — not bolted on by a separate turn-detector.

How it works (in one idea)

The LoRA sits only on the language model's attention/MLP projections; the audio encoder and embeddings stay frozen — we teach behavior, not perception.

The key finding behind this release: opening up, shutting up (on barge-in), and staying in a multi-turn conversation are the same problem — the "act now" signal is a tiny fraction of the supervised positions and gets drowned out by all the <wait> frames. Amplifying the training loss on those few decisive positions cures all three at once, with plain supervised fine-tuning.

Performance

Bare model, all runtime heuristics off (no VAD, no nudge) — this is the model on its own:

With a light runtime layer (energy-gated barge, silence-freeze), the live experience is smoother still — see the demo repo.

Usage

This is a PEFT LoRA adapter for google/gemma-4-E2B-it.

from peft import PeftModel
from transformers import Gemma4ForConditionalGeneration, AutoProcessor

base = Gemma4ForConditionalGeneration.from_pretrained("google/gemma-4-E2B-it")
model = PeftModel.from_pretrained(base, "dockhardman/gemma-4-E2B-duplex")
processor = AutoProcessor.from_pretrained("google/gemma-4-E2B-it")

The adapter runs frame-synchronously (80ms frames), so it needs a streaming driver to feed audio and drain text/tool events. A companion demo repo with a ready-to-run web app (microphone UI

• TTS output): https://github.com/allen2c/gemma-4-E2B-duplex

Limitations

• Semantics are bounded by a 2B base — great at conversational behavior, not a knowledge oracle.
• Barge-in isn't instant — the model shortens its tail to a few characters; a runtime energy gate handles hard cut-off. Native stop is strong but not zero-latency.
• Text-out only — you bring your own TTS.
• zh / en only.

License & attribution

• License: Apache-2.0 (inherited from the gemma-4-E2B-it base).
• Base model: google/gemma-4-E2B-it
• Author: allen2c

@misc{gemma4e2bduplex2026,
  author = {allen2c},
  title  = {Gemma-4-E2B-Duplex: a frame-synchronous full-duplex speech LoRA},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/dockhardman/gemma-4-E2B-duplex}}
}