spanish_conversational_tts

Spanish TTS Multiple Voices

This repository contains a fine-tuned Spanish Text-to-Speech (TTS) model based on canopylabs/3b-es_it-pretrain-research_release. The model supports multiple voices and nuanced emotions, trained using Unsloth and SNAC for audio tokenization.

➡️ Try it online: https://huggingface.co/spaces/sirekist98/spanish\ conversational\ tts

---

👨‍💻 Model Summary

• Base model: canopylabs/3b-es_it-pretrain-research_release
• Fine-tuned with: LoRA adapters (64 rank, alpha 64)
• Audio tokenization: SNAC (24kHz)
• Input format: source: text
• Dataset: ~6k samples, 9 speakers
• Training framework: Unsloth + Hugging Face Transformers

source: text

This prompt was then used to generate audio tokens.

We trained the model for 1 epoch using gradient accumulation (batch size 8 × 4 steps) with 4-bit quantization on an NVIDIA L4 GPU.

---

🔊 Inference

You can run inference using the demo space: Orpheus TTS Spanish Fine-Tuned.

To run inference locally with full control:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
from snac import SNAC

# --- Minimal config ---
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
BASE  = "canopylabs/3b-es_it-pretrain-research_release"
LORA  = "sirekist98/spanish_conversational_tts"
SNAC_ID = "hubertsiuzdak/snac_24khz"

VOICE = "Lucía"
TEXT = "Cuando voy al cine lo que más me suele gustar es comer las palomintas."
prompt = f"{VOICE}: {TEXT}"

# --- Load models ---
tokenizer  = AutoTokenizer.from_pretrained(BASE)
base_model = AutoModelForCausalLM.from_pretrained(
    BASE,
    torch_dtype=torch.float16 if device.type == "cuda" else torch.float32
)
model      = PeftModel.from_pretrained(base_model, LORA).to(device).eval()
snac_model = SNAC.from_pretrained(SNAC_ID).to(device)

# --- Prepare input (same as your Space) ---
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
start_tok = torch.tensor([[128259]], dtype=torch.long).to(device)
end_toks  = torch.tensor([[128009, 128260]], dtype=torch.long).to(device)

input_ids = torch.cat([start_tok, input_ids, end_toks], dim=1)
MAX_LEN   = 4260
pad_len   = MAX_LEN - input_ids.shape[1]
pad       = torch.full((1, pad_len), 128263, dtype=torch.long).to(device)
input_ids = torch.cat([pad, input_ids], dim=1)
attention_mask = torch.cat(
    [torch.zeros((1, pad_len), dtype=torch.long),
     torch.ones((1, input_ids.shape[1] - pad_len), dtype=torch.long)],
    dim=1
).to(device)

# --- Generate ---
generated = model.generate(
    input_ids=input_ids,
    attention_mask=attention_mask,
    max_new_tokens=1200,
    do_sample=True,
    temperature=0.6,
    top_p=0.95,
    repetition_penalty=1.1,
    num_return_sequences=1,
    eos_token_id=128258,
    use_cache=True
)

# --- Post-process (find 128257, remove 128258, multiple of 7, subtract 128266) ---
AUDIO_TOKEN_OFFSET = 128266
token_to_find      = 128257
token_to_remove    = 128258

idxs = (generated == token_to_find).nonzero(as_tuple=True)
cropped = generated[:, idxs[1][-1].item() + 1:] if len(idxs[1]) > 0 else generated
cleaned = cropped[cropped != token_to_remove]
codes   = cleaned[: (len(cleaned) // 7) * 7].tolist()
codes   = [int(t) - AUDIO_TOKEN_OFFSET for t in codes]

# --- SNAC decode (same layout as your Space) ---
layer_1, layer_2, layer_3 = [], [], []
for i in range((len(codes) + 1) // 7):
    b = 7 * i
    if b + 6 >= len(codes):
        break
    layer_1.append(codes[b + 0])
    layer_2.append(codes[b + 1] - 4096)
    layer_3.append(codes[b + 2] - 2 * 4096)
    layer_3.append(codes[b + 3] - 3 * 4096)
    layer_2.append(codes[b + 4] - 4 * 4096)
    layer_3.append(codes[b + 5] - 5 * 4096)
    layer_3.append(codes[b + 6] - 6 * 4096)

dev_snac = snac_model.quantizer.quantizers[0].codebook.weight.device
layers = [
    torch.tensor(layer_1).unsqueeze(0).to(dev_snac),
    torch.tensor(layer_2).unsqueeze(0).to(dev_snac),
    torch.tensor(layer_3).unsqueeze(0).to(dev_snac),
]

with torch.no_grad():
    audio = snac_model.decode(layers).squeeze().cpu().numpy()

# 'audio' is the 24kHz waveform.
# Optional:
# from scipy.io.wavfile import write as write_wav
# write_wav("output.wav", 24000, audio)

---

🗣️ Available Voices

You can generate speech using the following voices (source):

Alex, Carmen, Daniel, Diego, Hugo, Lucía, María, Pablo, Sofía

📖 Citation

@misc{sirekist2025spanishTTS,
  author = {sirekist98},
  title = {Spanish TTS Model with Emotions and Multiple Voices},
  year = {2025},
  howpublished = {\url{https://huggingface.co/sirekist98/spanish_model}}
}

---

✨ Acknowledgements

• Unsloth
• SNAC
• Hugging Face Datasets and Spaces

---

❓ Questions or Contributions?

Open an issue or contact @sirekist98 on Hugging Face.

Thanks for checking out this model! 🚀