Downloads · 30 days
127
68% of all-time downloads
furkanbekmezci/TR-Omni-E4B-Turkish-Native-Speech-to-Speech-Model
TR-Omni-E4B-Turkish-Native-Speech-to-Speech-Model is a any-to-any model from furkanbekmezci. Use it for the any-to-any task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
<p align="center" <img src="assets/overview-2.webp" width="100%" alt="TR-Omni-E4B, Turkish native speech-to-speech model. Inputs: Turkish speech at 16 kHz, an optional image and optional text, for example the spoken q…
Downloads · 30 days
127
68% of all-time downloads
All-time downloads
188
Public
Repo size
20.9 GB
Likes
4
Public
Click a slice to open those files.
.safetensors20.5 GB · 98%
From the Hugging Face model README
TR-Omni-E4B is a two-component (thinker–talker) Turkish speech-to-speech model. The thinker, Gemma 4 E4B adapted to Turkish speech, takes speech, images and text as input and writes the reply; speech is encoded directly, without an intermediate ASR transcript. The talker, initialised from Trendyol-TTS (VoxCPM2), generates 48 kHz speech one sentence at a time, conditioned on the thinker's sampled tokens together with its hidden states for them. The real-time browser app (sentence-level streaming, turn-taking, barge-in) is on GitHub: Furkansz/TR-Omni-E4B.
Positioning. The input side is native: the thinker encodes speech itself, with no ASR step. The output side follows the thinker–talker design of Qwen2.5-Omni and LLaMA-Omni2: a separate talker, here initialised from a TTS model, reads the reply's tokens together with the thinker's contextual hidden states. TR-Omni-E4B is therefore neither a cascade that hands only text to an independent TTS, nor a single model that generates audio directly (as GPT-4o or Moshi do). Pronunciation comes mainly from the tokens; whether the hidden states improve prosody has not been measured yet (see Limitations).
Türkçe — TR-Omni-E4B, iki bileşenli (thinker–talker) bir Türkçe speech-to-speech modelidir. Thinker (Türkçe konuşmaya uyarlanmış Gemma 4 E4B) konuşma, görüntü ve metin girdilerini alır ve cevabı yazar; konuşma girdisi ara bir ASR transkriptine dönüştürülmeden doğrudan işlenir. Trendyol-TTS'ten (VoxCPM2) başlatılan talker ise 48 kHz konuşmayı cümle cümle üretir ve bunu thinker'ın ürettiği token'larla birlikte bu token'lara ait hidden state'leri kullanarak yapar. Yani girdi tarafı native'dir; çıkış tarafı ise Qwen2.5-Omni ve LLaMA-Omni2'deki thinker–talker yapısını izler. Model, yalnızca metni bağımsız bir TTS'e aktaran bir cascade değildir; GPT-4o veya Moshi gibi sesi doğrudan üreten tek bir model de değildir. Tarayıcıda canlı sesli sohbet uygulaması GitHub'dadır: Furkansz/TR-Omni-E4B.
tr_omni.py returns the
audio after the full reply.| Input | speech (16 kHz), image, text; any combination |
| Output | text and speech (48 kHz), generated per sentence |
| Language | Turkish |
| Parameters | thinker 8 B with embeddings, 4.5 B effective (Gemma 4 E4B); talker 2.3 B (VoxCPM2) |
| Voice | a single synthetic voice (see Responsible use) |
| Throughput | ≈ 54 tokens/s thinker decoding on a 2g MIG slice of an H200 |
The thinker generates the reply greedily. Each completed sentence is normalised to its spoken form and re-encoded in context (system prompt, the last 4 s of the user turn and the reply so far), which yields hidden states from layers 14, 28 and 42, i.e. 3 × 2560 per thinker token. Each talker token is aligned to the thinker token containing its first character. A StateMapper (softmax layer mixing, LayerNorm, 2560 → 1024 projection with piece-position embeddings, MLP to 2048) maps the states into the talker's text-embedding space, where a per-dimension gate mixes them with the talker's own token embeddings (mean g ≈ 0.4). The talker, consisting of a text-semantic LM, a residual acoustic LM, a local diffusion transformer and an AudioVAE decoder, continues from a fixed synthetic voice prompt. Token conditioning keeps pronunciation exact; states alone preserve content but not orthography (see the ablation below). This follows the hidden-state and token fusion used by Qwen2.5-Omni and LLaMA-Omni2.
All results are author-run on the released weights with greedy decoding; sample sizes are given per test. Generated speech is scored by transcribing it with Whisper large-v3 and comparing the transcript with the intended text. The baseline is the unmodified Gemma 4 E4B-it given identical inputs.
Identical prompt for both models ("Bu ses kaydını yazıya dök."). Normalisation: Turkish-aware lower-casing (I/ı), numbers spelled out, punctuation removed, apostrophes dropped.
| Official test split | TR-Omni-E4B | Gemma 4 E4B-it |
|---|---|---|
| FLEURS tr_tr test (all 743 utterances) | 5.7 / 1.9 | 5.6 / 1.9 |
| Common Voice 17.0 tr test (1,000 random utterances, seed 0) | 10.2 / 3.2 | 14.8 / 5.7 |
| YODAS tr, 99 held-out in-the-wild clips (simplified normalisation, WER only) | 30.8 | 37.6 |
The YODAS row uses a different normalisation and is not directly comparable with the rows above. Without apostrophe removal, FLEURS and Common Voice give 7.2 vs 6.7 and 10.8 vs 15.4; TR-Omni tends to insert apostrophes after common nouns.
100 sentences from the FLEURS tr_tr test split (unseen in training), each generated as the reply to a spoken request ("Bana bir şey anlatır mısın?") with in-context states, as in live use.
| Talker conditioning | WER % | CER % |
|---|---|---|
| tokens + states (released) | 3.0 | 0.54 |
| states only | 36.4 | 13.6 |
| tokens only: Trendyol-TTS, the initialisation | 2.1 | 0.34 |
Token + state conditioning stays within 0.2 CER points of the TTS it was initialised from; residual errors concentrate on foreign proper nouns ("pohutukawa", "Dunlap") and occasional elided syllables. The ablation shows that the tokens anchor lexical accuracy. It does not show that the states improve intelligibility over text-only TTS; their intended role is to carry conversational context into the talker.
116 general-knowledge questions in Turkish (12 topics; 40 easy, 48 medium, 28 hard), synthesised with voices unseen in training. Answers are judged against short references by Gemma-4-31B-it (binary).
| Accuracy % | all | easy | medium | hard |
|---|---|---|---|---|
| TR-Omni-E4B, answer text | 78.4 | 90.0 | 81.3 | 57.1 |
| TR-Omni-E4B, answer as heard (Whisper transcript of the generated speech) | 71.6 | 85.0 | 75.0 | 46.4 |
| Gemma 4 E4B-it, answer text | 83.6 | 92.5 | 79.2 | 78.6 |
The gap on hard questions reflects knowledge loss from speech adaptation. Numbers are verbalised in the reply, which accounts for most arithmetic errors ("On iki çarpı on iki kaç eder?" → "Yüz altmış dört eder."). The "as heard" row also includes evaluator ASR errors ("on beş" → "415"); spoken output agrees with the answer text at 3.2 % WER / 0.9 % CER, and most of the remaining gap comes from symbols read letter by letter ("Au" → "a u", "H2O" → "he iki o").
40 synthetic images with known answers, 5 per skill (Turkish text reading, counting, highest and lowest bar, table lookup, colour, arithmetic, left/right). Each question is asked typed and spoken, the latter with the live system prompt.
| Correct (of 40) | typed question | spoken question |
|---|---|---|
| TR-Omni-E4B | 38 | 37 |
| Gemma 4 E4B-it | 38 | 39 |
Both models are error-free on text reading, charts, tables, colour and left/right; counting is the shared weakness (7 or 8 red circles among blue squares). The two spoken-question misses in arithmetic are verbalisation errors ("24" → "otuz dört"); the typed answers are correct.
| TR-Omni-E4B | Gemma 4 E4B-it | |
|---|---|---|
| speaker gender (Common Voice, FLEURS, VocalSound; n = 120) | 92.5 % | 19.2 % |
| vocal sounds: laugh, sigh, cough, throat clearing, sneeze, sniff (VocalSound; n = 29) | 96.6 % | 13.8 % |
| emotion, 6 classes (CREMA-D, English sentences; n = 19) | 36.8 % | 5.3 % |
Scored by label match; the base model rarely names a label. These are small diagnostic sets: emotion recognition is weak, and the model should not be used as an emotion or speaker-attribute classifier.
Download the full repository (thinker/, talker/, tr_omni.py, tr_speak.py) and run it from its root:
pip install torch "transformers>=5.17" voxcpm==2.0.3 soundfile numpy librosa pillow
python tr_omni.py --audio question.wav --out reply.wav # speech → speech
python tr_omni.py --audio question.wav --image photo.jpg --out reply.wav # speech + image → speech
python tr_omni.py --text "Kapadokya'yı anlatır mısın?" --out reply.wav # text → speech
from tr_omni import TROmni
omni = TROmni(".") # thinker/ and talker/
text, wav = omni.chat(audio=wav16k) # float32 16 kHz mono in, 48 kHz float32 out
text, wav = omni.chat(audio=wav16k, image=img) # img: PIL image
history = [] # multi-turn: pass the same list every turn
text, wav = omni.chat(audio=turn1, history=history)
text, wav = omni.chat(audio=turn2, history=history)
Live conversation in the browser. The streaming app (sentence-level streaming, Smart Turn v3.2 turn-taking, echo-cancelled barge-in in Chrome) is in a separate repository and downloads this model on its first run:
git clone https://github.com/Furkansz/TR-Omni-E4B.git && cd TR-Omni-E4B
pip install -r requirements.txt
python live/server.py # then open http://localhost:7860
| File | Content |
|---|---|
thinker/ | thinker weights (Gemma 4 E4B architecture, Transformers format) |
talker/ | talker in VoxCPM2 format (model.safetensors, audiovae.pth), native_adapter.safetensors (StateMapper, gate), native_config.json, voice/assistant.wav |
tr_omni.py | reference inference: thinker → per-sentence states → talker |
tr_speak.py | Turkish text normalisation for speech: numbers, ordinals (incl. Roman), dates, times, units, abbreviations, formulas |
Hardware. Measured peak: 15.4 GiB allocated by PyTorch, excluding the CUDA context and allocator overhead, so a
16 GB GPU may not be enough; 24 GB is comfortable. The thinker's 5.6 GB per-layer embedding table is kept in host
RAM. Decoding uses a static KV cache with one CUDA graph per token; the first call
includes about a minute of compilation. tr_omni.py is non-streaming: a two-sentence reply (9 s of audio) completes
14 s after the query on a 2g MIG slice of an H200. The ≈ 0.7 s first-audio latency refers to the sentence-level
streaming live app.
Thinker. Gemma 4 E4B-it, LoRA (r = 64) on the language model in two stages, merged into the released weights.
The vision tower and vision projector are unchanged.
Talker. Initialised from Trendyol-TTS (a Turkish VoxCPM2) and trained on ≈ 114 k Turkish utterances (YODAS, Common Voice 17, FLEURS train), with LoRA (r = 32) on the attention layers of the text-semantic LM, residual acoustic LM and local diffusion decoder, merged into the released weights.
tr_speak.py; rare abbreviations, foreign names and uncommon notation may be
misread.The assistant voice (talker/voice/assistant.wav) is synthetic, generated by Trendyol-TTS without a reference
speaker. The talker continues the voice of its prompt and can therefore imitate a voice from a short sample. In line
with the VoxCPM2 terms, do not use this model to impersonate real people, for fraud or for disinformation, and
label generated speech as AI-generated. Do not clone voices without consent.
Apache-2.0 (LICENSE); third-party notices in NOTICE.md. Built on
Gemma 4 E4B-it (Apache-2.0),
VoxCPM2 (Apache-2.0) and
Trendyol-TTS (MIT; trained by its authors on a private Turkish
speech dataset). Training data: Mozilla Common Voice 17.0 (CC0-1.0), Google FLEURS (CC-BY-4.0), MediaSpeech
(CC-BY-4.0), YODAS (CC-BY-3.0), Aya (Apache-2.0), VocalSound (CC-BY-SA-4.0), CREMA-D (ODbL-1.0), and synthetic text
and speech from Gemma 4 and VoxCPM2. Turn detection in the live app:
Smart Turn v3 (BSD-2-Clause).