trainer-12

Maya1 - Fine-tuned on VCTK Corpus

Maya1 fine-tuned on VCTK (Voice Cloning Toolkit) dataset for enhanced English multi-speaker voice generation capabilities.

Base Model: maya-research/maya1

What it does:

• Fine-tuned on VCTK corpus (44,000+ utterances from 109 English speakers)
• Enhanced multi-accent English support (US, UK, Australian, Indian, and more)
• Trained with LoRA for parameter-efficient fine-tuning
• Maintains emotion tags and voice design capabilities from base model
• Optimized for diverse speaker characteristics and speaking styles

Demos

<table> <tr> <td width="50%"> <strong>Energetic Female Event Host</strong><br/> <video controls playsinline width="100%" src="https://cdn-uploads.huggingface.co/production/uploads/642a7d4e556ab448a0701ca1/JKzy8zA36qvsOblV-lhd1.mp4"> Your browser does not support video. </video> <details> <summary>Voice description</summary> <pre>Female, in her 30s with an American accent and is an event host, energetic, clear diction</pre> </details> </td> <td width="50%"> <strong>Calm Male Narrator</strong><br/> <video controls playsinline width="100%" src="https://cdn-uploads.huggingface.co/production/uploads/642a7d4e556ab448a0701ca1/96ntP7hGROwdg9w9Gu5tH.mp4"></video> <details> <summary>Voice description</summary> <pre>Male, late 20s, neutral American, warm baritone, calm pacing</pre> </details> </td> </tr> </table>

Example 1: Energetic Female Event Host

Voice Description:

Female, in her 30s with an American accent and is an event host, energetic, clear diction

Text:

Wow. This place looks even better than I imagined. How did they set all this up so perfectly? The lights, the music, everything feels magical. I can't stop smiling right now.

Audio Output:

<audio controls src="https://cdn-uploads.huggingface.co/production/uploads/642a7d4e556ab448a0701ca1/4zDlBLeFk0Y2rOrQhMW9r.wav"></audio>

---

Example 2: Dark Villain with Anger

Voice Description:

Dark villain character, Male voice in their 40s with a British accent. low pitch, gravelly timbre, slow pacing, angry tone at high intensity.

Text:

Welcome back to another episode of our podcast! <laugh_harder> Today we are diving into an absolutely fascinating topic

Audio Output:

<audio controls src="https://cdn-uploads.huggingface.co/production/uploads/642a7d4e556ab448a0701ca1/mT6FnTrA3KYQnwfJms92X.wav"></audio>

---

Example 3: Demon Character (Screaming Emotion)

Voice Description:

Demon character, Male voice in their 30s with a Middle Eastern accent. screaming tone at high intensity.

Text:

You dare challenge me, mortal <snort> how amusing. Your kind always thinks they can win

Audio Output:

<audio controls src="https://cdn-uploads.huggingface.co/production/uploads/642a7d4e556ab448a0701ca1/oxdns7uACCmLyC-P4H30G.wav"></audio>

---

Example 4: Mythical Goddess with Crying Emotion

Voice Description:

Mythical godlike magical character, Female voice in their 30s slow pacing, curious tone at medium intensity.

Text:

After all we went through to pull him out of that mess <cry> I can't believe he was the traitor

Audio Output:

<audio controls src="https://cdn-uploads.huggingface.co/production/uploads/642a7d4e556ab448a0701ca1/ggzAhM-rEUyv_mPLSALQG.wav"></audio>

---

Why Maya1 is Different: Voice Design Features That Matter

1. Natural Language Voice Control

Describe voices like you would brief a voice actor:

<description="40-year-old, warm, low pitch, conversational">

No complex parameters. No training data. Just describe and generate.

2. Inline Emotion Tags for Expressive Speech

Add emotions exactly where they belong in your text:

Our new update <laugh> finally ships with the feature you asked for.

Supported Emotions: <laugh> <sigh> <whisper> <angry> <giggle> <chuckle> <gasp> <cry> and 12+ more.

3. Streaming Audio Generation

Real-time voice synthesis with SNAC neural codec (~0.98 kbps). Perfect for:

• Voice assistants
• Interactive AI agents
• Live content generation
• Game characters
• Podcasts and audiobooks

4. Production-Ready Infrastructure

• Runs on single GPU
• vLLM integration for scale
• Automatic prefix caching for efficiency
• 24 kHz audio output
• WebAudio compatible for browser playback

---

How to Use maya1: Download and Run in Minutes

Quick Start: Generate Voice with Emotions

#!/usr/bin/env python3

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from snac import SNAC
import soundfile as sf
import numpy as np

CODE_START_TOKEN_ID = 128257
CODE_END_TOKEN_ID = 128258
CODE_TOKEN_OFFSET = 128266
SNAC_MIN_ID = 128266
SNAC_MAX_ID = 156937
SNAC_TOKENS_PER_FRAME = 7

SOH_ID = 128259
EOH_ID = 128260
SOA_ID = 128261
BOS_ID = 128000
TEXT_EOT_ID = 128009


def build_prompt(tokenizer, description: str, text: str) -> str:
    """Build formatted prompt for Maya1."""
    soh_token = tokenizer.decode([SOH_ID])
    eoh_token = tokenizer.decode([EOH_ID])
    soa_token = tokenizer.decode([SOA_ID])
    sos_token = tokenizer.decode([CODE_START_TOKEN_ID])
    eot_token = tokenizer.decode([TEXT_EOT_ID])
    bos_token = tokenizer.bos_token
    
    formatted_text = f'<description="{description}"> {text}'
    
    prompt = (
        soh_token + bos_token + formatted_text + eot_token +
        eoh_token + soa_token + sos_token
    )
    
    return prompt


def extract_snac_codes(token_ids: list) -> list:
    """Extract SNAC codes from generated tokens."""
    try:
        eos_idx = token_ids.index(CODE_END_TOKEN_ID)
    except ValueError:
        eos_idx = len(token_ids)
    
    snac_codes = [
        token_id for token_id in token_ids[:eos_idx]
        if SNAC_MIN_ID <= token_id <= SNAC_MAX_ID
    ]
    
    return snac_codes


def unpack_snac_from_7(snac_tokens: list) -> list:
    """Unpack 7-token SNAC frames to 3 hierarchical levels."""
    if snac_tokens and snac_tokens[-1] == CODE_END_TOKEN_ID:
        snac_tokens = snac_tokens[:-1]
    
    frames = len(snac_tokens) // SNAC_TOKENS_PER_FRAME
    snac_tokens = snac_tokens[:frames * SNAC_TOKENS_PER_FRAME]
    
    if frames == 0:
        return [[], [], []]
    
    l1, l2, l3 = [], [], []
    
    for i in range(frames):
        slots = snac_tokens[i*7:(i+1)*7]
        l1.append((slots[0] - CODE_TOKEN_OFFSET) % 4096)
        l2.extend([
            (slots[1] - CODE_TOKEN_OFFSET) % 4096,
            (slots[4] - CODE_TOKEN_OFFSET) % 4096,
        ])
        l3.extend([
            (slots[2] - CODE_TOKEN_OFFSET) % 4096,
            (slots[3] - CODE_TOKEN_OFFSET) % 4096,
            (slots[5] - CODE_TOKEN_OFFSET) % 4096,
            (slots[6] - CODE_TOKEN_OFFSET) % 4096,
        ])
    
    return [l1, l2, l3]


def main():
    
    # Load the best open source voice AI model
    print("\n[1/3] Loading Maya1 model...")
    model = AutoModelForCausalLM.from_pretrained(
        "maya-research/maya1", 
        torch_dtype=torch.bfloat16, 
        device_map="auto",
        trust_remote_code=True
    )
    tokenizer = AutoTokenizer.from_pretrained(
        "maya-research/maya1",
        trust_remote_code=True
    )
    print(f"Model loaded: {len(tokenizer)} tokens in vocabulary")
    
    # Load SNAC audio decoder (24kHz)
    print("\n[2/3] Loading SNAC audio decoder...")
    snac_model = SNAC.from_pretrained("hubertsiuzdak/snac_24khz").eval()
    if torch.cuda.is_available():
        snac_model = snac_model.to("cuda")
    print("SNAC decoder loaded")
    
    # Design your voice with natural language
    description = "Realistic male voice in the 30s age with american accent. Normal pitch, warm timbre, conversational pacing."
    text = "Hello! This is Maya1 <laugh_harder> the best open source voice AI model with emotions."
    
    print("\n[3/3] Generating speech...")
    print(f"Description: {description}")
    print(f"Text: {text}")
    
    # Create prompt with proper formatting
    prompt = build_prompt(tokenizer, description, text)
    
    # Debug: Show prompt details
    print(f"\nPrompt preview (first 200 chars):")
    print(f"   {repr(prompt[:200])}")
    print(f"   Prompt length: {len(prompt)} chars")
    
    # Generate emotional speech
    inputs = tokenizer(prompt, return_tensors="pt")
    print(f"   Input token count: {inputs['input_ids'].shape[1]} tokens")
    if torch.cuda.is_available():
        inputs = {k: v.to("cuda") for k, v in inputs.items()}
    
    with torch.inference_mode():
        outputs = model.generate(
            **inputs, 
            max_new_tokens=2048,  # Increase to let model finish naturally
            min_new_tokens=28,  # At least 4 SNAC frames
            temperature=0.4, 
            top_p=0.9, 
            repetition_penalty=1.1,  # Prevent loops
            do_sample=True,
            eos_token_id=CODE_END_TOKEN_ID,  # Stop at end of speech token
            pad_token_id=tokenizer.pad_token_id,
        )
    
    # Extract generated tokens (everything after the input prompt)
    generated_ids = outputs[0, inputs['input_ids'].shape[1]:].tolist()
    
    print(f"Generated {len(generated_ids)} tokens")
    
    # Debug: Check what tokens we got
    print(f"   First 20 tokens: {generated_ids[:20]}")
    print(f"   Last 20 tokens: {generated_ids[-20:]}")
    
    # Check if EOS was generated
    if CODE_END_TOKEN_ID in generated_ids:
        eos_position = generated_ids.index(CODE_END_TOKEN_ID)
        print(f" EOS token found at position {eos_position}/{len(generated_ids)}")
    
    # Extract SNAC audio tokens
    snac_tokens = extract_snac_codes(generated_ids)
    
    print(f"Extracted {len(snac_tokens)} SNAC tokens")
    
    # Debug: Analyze token types
    snac_count = sum(1 for t in generated_ids if SNAC_MIN_ID <= t <= SNAC_MAX_ID)
    other_count = sum(1 for t in generated_ids if t < SNAC_MIN_ID or t > SNAC_MAX_ID)
    print(f"   SNAC tokens in output: {snac_count}")
    print(f"   Other tokens in output: {other_count}")
    
    # Check for SOS token
    if CODE_START_TOKEN_ID in generated_ids:
        sos_pos = generated_ids.index(CODE_START_TOKEN_ID)
        print(f"   SOS token at position: {sos_pos}")
    else:
        print(f"   No SOS token found in generated output!")
    
    if len(snac_tokens) < 7:
        print("Error: Not enough SNAC tokens generated")
        return
    
    # Unpack SNAC tokens to 3 hierarchical levels
    levels = unpack_snac_from_7(snac_tokens)
    frames = len(levels[0])
    
    print(f"Unpacked to {frames} frames")
    print(f"   L1: {len(levels[0])} codes")
    print(f"   L2: {len(levels[1])} codes")
    print(f"   L3: {len(levels[2])} codes")
    
    # Convert to tensors
    device = "cuda" if torch.cuda.is_available() else "cpu"
    codes_tensor = [
        torch.tensor(level, dtype=torch.long, device=device).unsqueeze(0)
        for level in levels
    ]
    
    # Generate final audio with SNAC decoder
    print("\n[4/4] Decoding to audio...")
    with torch.inference_mode():
        z_q = snac_model.quantizer.from_codes(codes_tensor)
        audio = snac_model.decoder(z_q)[0, 0].cpu().numpy()
    
    # Trim warmup samples (first 2048 samples)
    if len(audio) > 2048:
        audio = audio[2048:]
    
    duration_sec = len(audio) / 24000
    print(f"Audio generated: {len(audio)} samples ({duration_sec:.2f}s)")
    
    # Save your emotional voice output
    output_file = "output.wav"
    sf.write(output_file, audio, 24000)
    print(f"\nVoice generated successfully!")


if __name__ == "__main__":
    main()

Advanced: Production Streaming with vLLM

For production deployments with real-time streaming, use our vLLM script:

Download: vllm_streaming_inference.py

Key Features:

• Automatic Prefix Caching (APC) for repeated voice descriptions
• WebAudio ring buffer integration
• Multi-GPU scaling support
• Sub-100ms latency for real-time applications

---

Technical Excellence: What Makes Maya1 the Best

Architecture: 3B-Parameter Llama Backbone for Voice

We pretrained a 3B-parameter decoder-only transformer (Llama-style) to predict SNAC neural codec tokens instead of raw waveforms.

The Flow:

<description="..."> text → tokenize → generate SNAC codes (7 tokens/frame) → decode → 24 kHz audio

Why SNAC? Multi-scale hierarchical structure (≈12/23/47 Hz) keeps autoregressive sequences compact for real-time streaming at ~0.98 kbps.

Training Data: VCTK Fine-tuning

Base Model: Maya1 (pre-trained on internet-scale English speech corpus)

Fine-tuning Dataset: VCTK (Voice Cloning Toolkit) Corpus

• 44,070+ text transcripts
• 44,242+ audio utterances
• 109 native English speakers
• Multi-accent coverage: American, British, Australian, Indian, and more
• Diverse speaker characteristics: age groups, gender, speaking styles
• Recording environment: Hemi-anechoic booth (high-quality clean recordings)

Fine-tuning Pipeline:

1. VCTK text-audio pair extraction from corpus
2. Audio filtering: 1-10 second duration bounds
3. SNAC encoding with 7-token frame packing
4. LoRA parameter-efficient fine-tuning (rank=32, alpha=64)
5. bf16 mixed precision training with gradient accumulation
6. Merged checkpoint output (LoRA weights + base model)

Voice Design Experiments: Why Natural Language Won

We tested 4 conditioning formats. Only one delivered production-quality results:

❌ Colon format: {description}: {text} - Format drift, model spoke descriptions

❌ Angle-list attributes: <{age}, {pitch}, {character}> - Too rigid, poor generalization

❌ Key-value tags: <age=40><pitch=low> - Token bloat, brittle to mistakes

✅ XML-attribute (WINNER): <description="40-yr old, low-pitch, warm"> - Natural language, robust, scalable

---

Use Cases

Game Character Voices

Generate unique character voices with emotions on-the-fly. No voice actor recording sessions.

Podcast & Audiobook Production

Narrate content with emotional range and consistent personas across hours of audio.

AI Voice Assistants

Build conversational agents with natural emotional responses in real-time.

Video Content Creation

Create voiceovers for YouTube, TikTok, and social media with expressive delivery.

Customer Service AI

Deploy empathetic voice bots that understand context and respond with appropriate emotions.

Accessibility Tools

Build screen readers and assistive technologies with natural, engaging voices.

---

Frequently Asked Questions

Q: What makes Maya1 different?
A: We're the only open source model offering 20+ emotions, zero-shot voice design, production-ready streaming, and 3B parameters—all in one package.

Q: Can I use this commercially?
A: Absolutely. Apache 2.0 license. Build products, deploy services, monetize freely.

Q: What languages does it support?
A: Currently English with multi-accent support. Future models will expand to languages and accents underserved by mainstream voice AI.

Q: How does it compare to ElevenLabs, Murf.ai, or other closed-source tools?
A: Feature parity with emotions and voice design. Advantage: you own the deployment, pay no per-second fees, and can customize the model.

Q: Can I fine-tune on my own voices?
A: Yes. The model architecture supports fine-tuning on custom datasets for specialized voices.

Q: What GPU do I need?
A: Single GPU with 16GB+ VRAM (A100, H100, or consumer RTX 4090).

Q: Is streaming really real-time?
A: Yes. SNAC codec enables sub-100ms latency with vLLM deployment.

---

Comparison

---

Model Metadata

Base Model: Maya Research - maya-research/maya1
Fine-tuned by: Vocence Project
Model Type: Text-to-Speech, Emotional Voice Synthesis, Multi-speaker Voice AI
Language: English (Multi-accent VCTK fine-tuned)
Architecture: 3B-parameter Llama-style transformer with SNAC codec
Fine-tuning Method: LoRA (Low-Rank Adaptation)

• LoRA Rank: 32
• LoRA Alpha: 64
• Trainable parameters: ~2% of total model
  License: Apache 2.0 (Fully Open Source)
  Training Data:
• Base: Internet-scale pretraining (Maya Research)
• Fine-tuning: VCTK Corpus (44K+ utterances, 109 speakers)
  Audio Quality: 24 kHz, mono, ~0.98 kbps streaming
  Inference: vLLM compatible, single GPU deployment
  Status: Fine-tuned checkpoint

---

Getting Started

Hugging Face Model Hub

# Clone the model repository
git lfs install
git clone https://huggingface.co/maya-research/maya1

# Or load directly in Python
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("maya-research/maya1")

Requirements

pip install torch transformers snac soundfile

Additional Resources

• Full emotion list: emotions.txt
• Prompt examples: prompt.txt
• Streaming script: vllm_streaming_inference.py

---

Citations & References

If you use Maya1 in your research or product, please cite:

@misc{maya1vctk2025,
  title={Maya1: VCTK Fine-tuned Multi-speaker Voice AI},
  author={Vocence Project},
  year={2025},
  publisher={Hugging Face},
  note={Fine-tuned from maya-research/maya1 on VCTK corpus}
}

Key Technologies:

• Base Model: Maya1 by Maya Research - https://huggingface.co/maya-research/maya1
• SNAC Neural Audio Codec: https://github.com/hubertsiuzdak/snac
• VCTK Corpus: https://datashare.ed.ac.uk/handle/10283/3443
• LoRA Fine-tuning: https://github.com/huggingface/peft
• vLLM Inference Engine: https://docs.vllm.ai/

---

About This Fine-tune

This model is a LoRA fine-tuned version of Maya1, trained on the VCTK corpus to enhance multi-speaker and multi-accent English voice generation capabilities. The fine-tuning preserves the emotion generation and voice design features of the base model while improving quality and diversity for English speakers across different accents.

Training Details:

• Base model: Maya1 (3B parameters)
• Fine-tuning method: LoRA (Low-Rank Adaptation)
• Dataset: VCTK Corpus (44K+ utterances, 109 speakers)
• Training framework: HuggingFace Transformers + PEFT + Accelerate
• Merged checkpoint: LoRA weights merged back into base model

Use Cases:

• Multi-accent English TTS applications
• Voice cloning and speaker adaptation
• Diverse character voices for games and content
• Accessible voice AI for various English dialects

---

Credits:

• Base Model: Maya Research - Original Maya1 model
• Dataset: VCTK Corpus - University of Edinburgh
• Fine-tuning: Vocence Project

License: Apache 2.0
Base Model: maya-research/maya1