veena-hinglish

Veena Hinglish TTS

Fine-tuned Veena TTS for Hinglish (Hindi-English code-mixed) text-to-speech synthesis.

MOS: Base Veena 4.12/5 → Fine-tuned 4.66/5

Quick Start

Installation

pip install transformers torch snac soundfile
pip install bitsandbytes  # optional, for 4-bit quantized inference

Inference

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from snac import SNAC
import soundfile as sf

# --- Load models ---
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "akh-mysterio/veena-hinglish",
    quantization_config=quantization_config,   # remove for full fp16
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("akh-mysterio/veena-hinglish", trust_remote_code=True)
snac_model = SNAC.from_pretrained("hubertsiuzdak/snac_24khz").eval().cuda()

# --- Control tokens ---
START_OF_SPEECH_TOKEN = 128257
END_OF_SPEECH_TOKEN   = 128258
START_OF_HUMAN_TOKEN  = 128259
END_OF_HUMAN_TOKEN    = 128260
START_OF_AI_TOKEN     = 128261
END_OF_AI_TOKEN       = 128262
AUDIO_CODE_BASE_OFFSET = 128266


def decode_snac_tokens(snac_tokens, snac_model):
    """De-interleave and decode SNAC tokens to audio waveform."""
    if not snac_tokens or len(snac_tokens) % 7 != 0:
        return None

    snac_device = next(snac_model.parameters()).device
    offsets = [AUDIO_CODE_BASE_OFFSET + i * 4096 for i in range(7)]

    codes = [[] for _ in range(3)]
    for i in range(0, len(snac_tokens), 7):
        codes[0].append(snac_tokens[i]     - offsets[0])       # coarse
        codes[1].append(snac_tokens[i + 1] - offsets[1])       # medium
        codes[1].append(snac_tokens[i + 4] - offsets[4])
        codes[2].append(snac_tokens[i + 2] - offsets[2])       # fine
        codes[2].append(snac_tokens[i + 3] - offsets[3])
        codes[2].append(snac_tokens[i + 5] - offsets[5])
        codes[2].append(snac_tokens[i + 6] - offsets[6])

    hierarchical = [
        torch.tensor(c, dtype=torch.int32, device=snac_device).unsqueeze(0)
        for c in codes
    ]
    with torch.no_grad():
        audio_hat = snac_model.decode(hierarchical)
    return audio_hat.squeeze().clamp(-1, 1).cpu().numpy()


def generate_speech(text, speaker="kavya", temperature=0.4, top_p=0.9):
    """Generate speech audio from text."""
    prompt = f"<spk_{speaker}> {text}"
    prompt_tokens = tokenizer.encode(prompt, add_special_tokens=False)

    input_tokens = [
        START_OF_HUMAN_TOKEN,
        *prompt_tokens,
        END_OF_HUMAN_TOKEN,
        START_OF_AI_TOKEN,
        START_OF_SPEECH_TOKEN,
    ]
    input_ids = torch.tensor([input_tokens], device=model.device)
    max_tokens = min(int(len(text) * 1.3) * 7 + 21, 700)

    with torch.no_grad():
        output = model.generate(
            input_ids,
            max_new_tokens=max_tokens,
            do_sample=True,
            temperature=temperature,
            top_p=top_p,
            repetition_penalty=1.05,
            pad_token_id=tokenizer.pad_token_id,
            eos_token_id=[END_OF_SPEECH_TOKEN, END_OF_AI_TOKEN],
        )

    generated_ids = output[0][len(input_tokens):].tolist()
    snac_tokens = [
        t for t in generated_ids
        if AUDIO_CODE_BASE_OFFSET <= t < (AUDIO_CODE_BASE_OFFSET + 7 * 4096)
    ]
    return decode_snac_tokens(snac_tokens, snac_model)


# --- Generate ---
audio = generate_speech(
    "Aaj mausam bohot acha hai, chalo bahar chalte hain!",
    speaker="kavya",
)
if audio is not None:
    sf.write("output.wav", audio, 24000)

Model Details

Training

Fine-tuned from maya-research/Veena using LoRA on Hinglish speech data.

• Dataset: akh99/indictts-hinglish — IndicTTS Hindi corpus converted to Hinglish using LLM transliteration
• Method: LoRA adaptation, then merged into base weights
• Hardware: NVIDIA H100 80GB

Why Hinglish?

Hindi-English code-mixing ("Hinglish") is the dominant spoken register across urban India. The base Veena model handles Hindi and English separately but struggles with natural code-switching. This fine-tune bridges that gap.

Evaluation

+13% relative improvement in perceived speech quality on Hinglish evaluation set.

Source Code

Full training, inference, and streaming code: github.com/adola700/vee-ana

Acknowledgments

• Base model: Maya Research - Veena TTS
• Audio codec: SNAC