orpheus-3b-persian-tts-lora

Orpheus-3B Persian TTS (LoRA)

A LoRA adapter that adapts unsloth/orpheus-3b-0.1-ft — an English speech-LM built on Llama-3.2-3B that emits SNAC audio codec tokens — to Persian (Farsi) text-to-speech.

The adapter takes Finglish input: Persian romanized into Latin script (salam, hale shoma chetore?), not Persian script (سلام، حال شما چطوره؟). This was the deliberate design choice of the project — it reuses the base model's existing Latin-script tokenization instead of forcing it to learn an unseen writing system from ~50 hours of audio. See Input format.

Results

Evaluated against 1,219 held-out sentences (in-domain + challenge sets), transcribed with Whisper for ASR-based error rates. Lower is better for CER/WER/RTF.

Split by evaluation set, CER is 0.103 in-domain and 0.267 on the challenge set. Failure rate was 0.0% — every one of the 1,219 prompts produced decodable audio.

Two things worth reading carefully:

• The model scores below the SNAC codec reconstruction floor (0.129 vs 0.144). That does not mean it beats ground-truth audio; it means the codec's own reconstruction of the reference is harder for Whisper to transcribe than the model's clean synthetic speech. Treat 0.144 as the practical ceiling of this token protocol, not as a baseline the model surpassed in any meaningful sense.
• RTF 1.50 means it is slower than real-time on an RTX 3090 Ti — roughly 1.5 s of compute per second of audio. mms-tts-fas is ~150× faster. This is an autoregressive 3B LM emitting 7 codec tokens per 85 ms frame; it is not suited to low-latency streaming without further work.

Speaker similarity is measured only on the single-speaker in-domain set. Persian LID is the fraction of outputs Whisper identifies as Persian.

Input format

The model was trained on Finglish and expects Finglish. Feeding it Persian script is an out-of-distribution condition that the source project measured as a separate, confounded ablation (see Limitations).

The prompt protocol is inherited from Orpheus — a voice tag, then the text, wrapped in special tokens:

[128259] "female1: <finglish text>" [128009] [128260]

female1 is the only voice in the training corpus; other tags are untrained.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
from snac import SNAC

ADAPTER = "hofarah/orpheus-3b-persian-tts-lora"
BASE = "unsloth/orpheus-3b-0.1-ft"

# The adapter repo carries the resized tokenizer (base vocab + audio tokens),
# so load the tokenizer from here, not from the base model.
tok = AutoTokenizer.from_pretrained(ADAPTER)
model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16, device_map="cuda")
if len(tok) > model.get_input_embeddings().weight.shape[0]:
    model.resize_token_embeddings(len(tok))
model = PeftModel.from_pretrained(model, ADAPTER).eval()
snac = SNAC.from_pretrained("hubertsiuzdak/snac_24khz").eval()

START_OF_HUMAN, END_OF_TEXT, END_OF_HUMAN = 128259, 128009, 128260
START_OF_SPEECH, END_OF_SPEECH, AUDIO_BASE = 128257, 128258, 128266

text = "salam, hale shoma chetore?"          # Finglish, not Persian script
ids = tok(f"female1: {text}", return_tensors="pt").input_ids
ids = torch.cat([torch.tensor([[START_OF_HUMAN]]), ids,
                 torch.tensor([[END_OF_TEXT, END_OF_HUMAN]])], dim=1).to(model.device)

with torch.inference_mode():
    out = model.generate(input_ids=ids, attention_mask=torch.ones_like(ids),
                         max_new_tokens=1200, do_sample=True, temperature=0.6,
                         top_p=0.95, repetition_penalty=1.1,
                         eos_token_id=END_OF_SPEECH)

# Crop to the generated speech span, then de-interleave 7-token groups into
# SNAC's three codebook layers (L1: slot 0; L2: slots 1,4; L3: slots 2,3,5,6).
row = out[0].cpu()
hits = (row == START_OF_SPEECH).nonzero()
row = row[int(hits[-1]) + 1:] if hits.numel() else row
row = row[row != END_OF_SPEECH]
row = row[row >= AUDIO_BASE] - AUDIO_BASE
row = row[: (row.numel() // 7) * 7]

l1, l2, l3 = [], [], []
for i in range(row.numel() // 7):
    g = row[i * 7:(i + 1) * 7].tolist()
    if all(0 <= v < 4096 for v in (g[0], g[1] - 4096, g[4] - 4 * 4096,
                                   g[2] - 2 * 4096, g[3] - 3 * 4096,
                                   g[5] - 5 * 4096, g[6] - 6 * 4096)):
        l1 += [g[0]]
        l2 += [g[1] - 4096, g[4] - 4 * 4096]
        l3 += [g[2] - 2 * 4096, g[3] - 3 * 4096, g[5] - 5 * 4096, g[6] - 6 * 4096]

codes = [torch.tensor(x).unsqueeze(0) for x in (l1, l2, l3)]
with torch.inference_mode():
    wav = snac.decode(codes).squeeze().float().numpy()   # 24 kHz mono

A group containing an out-of-range index is a codec failure — drop it and count it rather than clamping, so the rate stays measurable.

Training

Inference was evaluated in bf16 rather than 4-bit, because the LoRA was trained against an unquantized base — bf16 is closer to the training condition.

trainer_state.json (full loss curve) is kept in this repo. Optimizer, scheduler and RNG state were removed; this repo is for inference, not for resuming training.

Limitations

• Single speaker. One voice (female1) in the entire corpus. There is no speaker control, and the speaker-similarity number describes consistency with that one voice.
• ≈13% of the training text is corrupted. A bare except: in the corpus's Finglish generation script wrote raw Persian text into the Finglish column whenever the transliteration API call failed. The composition is 86.73% clean Finglish, 10.46% pure Persian script, 2.04% mixed, 0.77% empty/single-char. Consequence: the model saw ~2,400 Persian-script examples, so any apparent Persian-script ability has a direct training-data explanation and is not zero-shot cross-script transfer.
• Length cap. Generation stops at 1,200 tokens ≈ 14.6 s. 15.9% of training utterances were longer than this, so long inputs will truncate.
• Slower than real-time (RTF 1.50), see Results.
• Corpus size is ≈50.8 h, verified from the released artifacts. Earlier project notes claiming ~29 h or "over 20 hours" are not reproducible and should not be cited.
• Evaluation reference audio for the in-domain set is SNAC codec reconstruction, not original waveform.
• MOS figures in the source evaluation are predicted MOS, not human listening-test MOS.

License

Not yet set. This is a LoRA adapter over unsloth/orpheus-3b-0.1-ft, which derives from Llama-3.2-3B — any license here must be compatible with the Llama 3.2 Community License and with the terms of the underlying speech corpus. Confirm both before citing or redistributing.

Citation

@misc{orpheus_persian_tts_lora,
  title  = {Orpheus-3B Persian TTS (LoRA)},
  author = {hofarah},
  year   = {2026},
  url    = {https://huggingface.co/hofarah/orpheus-3b-persian-tts-lora}
}