scenema-audio

Scenema Audio

Zero-shot expressive voice cloning and speech generation.

Visit scenema.ai/audio to hear all demos and try it out.

Watch the demo video on YouTube

Every existing text-to-speech system converts words into sound, but none of them perform. Scenema Audio generates speech with intention, pacing, breath control, and emotional arcs that shift within a single generation, all from a text prompt that describes not just what to say but how to say it.

Built on an audio diffusion transformer extracted from LTX 2.3's 22B parameter audiovisual model, it learned how people actually sound in real scenes: angry, laughing, whispering, crying, exhausted, terrified.

Capabilities

• Emotional acting: Rage, grief, joy, fear, exhaustion. Emotional state shifts within a single generation via action tags.
• Child voices: Six-year-olds, toddlers, teenagers. Naturally voiced, not pitch-shifted adults.
• Scene-aware audio: Describe the environment and the model generates speech with rain, thunder, crowds, or any ambient audio alongside the voice.
• Zero-shot voice cloning: Provide 10-20 seconds of reference audio with some emotional variability. The model transfers the voice identity onto any emotional performance. No fine-tuning, no enrollment.
• Long-form narration: Generates any length of audio by automatically splitting text and maintaining voice continuity across segments.
• Multilingual: English, German, French, Spanish, Italian, Portuguese, Japanese, Chinese, Korean, Russian, Arabic, Hindi, Swahili.

Model Checkpoints

Quick Start

git clone https://github.com/ScenemaAI/scenema-audio.git
cd scenema-audio

export HF_TOKEN=your_huggingface_token
docker compose up

Models are downloaded on first start (~38 GB) and cached in a Docker volume. See the GitHub repo for full documentation.

Prompt Format

<speak voice="VOICE_DESCRIPTION" gender="male|female"
       scene="OPTIONAL_SCENE" language="OPTIONAL_LANG_CODE">
  <action>Performance direction.</action>
  Speech text here.
</speak>

Voice Description

The voice attribute is the primary control. The richer and more specific, the better:

• Vocal qualities: timbre, pitch, breathiness, rasp, resonance
• Emotional state: rage, tenderness, exhaustion, excitement, grief
• Speaking style: pacing, emphasis, pauses, enunciation
• Character archetypes: "Think Tony Soprano having a breakdown"
• Age and gender: child, elderly, young woman, teenage boy
• Accents: British, Southern American, New Jersey Italian American

Action Tags

<action> tags are stage directions that shape HOW speech is delivered. Place them between speech segments to direct emotional shifts, pacing, and physical delivery:

<speak voice="Middle-aged man, warm but weathered." gender="male">
  <action>Calm, almost casual. Staring at his hands.</action>
  I used to think I had all the time in the world.
  <action>Voice tightens. Fighting to stay composed.</action>
  Then one Tuesday morning, the doctor said three words that changed everything.
  <action>Long pause. Deep breath. Raw but steady.</action>
  And I realized I hadn't called my son in six months.
</speak>

Voice Cloning

Provide 10-20 seconds of reference audio with some emotional variability. The model generates expressive speech from the prompt and transfers the reference voice's identity onto the performance.

{
  "prompt": "<speak voice=\"Gravelly male voice, fast talking, rough.\" gender=\"male\"><action>He completely loses it</action>What are you waiting for?!</speak>",
  "reference_voice_url": "https://example.com/reference.wav"
}

Any voice can perform any emotion, even if that voice has never been recorded in that emotional state.

Examples

Emotional Acting

<speak voice="A man on the edge. Explosive rage. Italian-American inflection."
       gender="male" scene="A dimly lit office, late at night">
  <action>He stands up slowly, voice dangerously low</action>
  You come into my house, you eat my food, and then you got the nerve
  to tell me how to run my business.
  <action>Voice rising, finger pointing</action>
  I built this thing from nothing while you were sitting on your ass.
</speak>

Child Voice

<speak voice="A six-year-old girl, bright and excited, speaking fast
with breathless enthusiasm. Slight lisp on S sounds."
gender="female">
  Mommy look! There is a rainbow and it goes all the way across the whole sky!
</speak>

Scene-Aware Audio

<speak voice="Male, mid 40s. Weathered. Urgent, projecting over wind."
       gender="male" scene="Open dock in a thunderstorm, heavy rain"
       shot="scene">
  <sound>Heavy rain and wind howling</sound>
  <action>He shouts over the storm</action>
  Get the lines! She is pulling loose!
  <sound>Thunder cracks overhead</sound>
  Move! I said move!
</speak>

API Reference

POST /generate

Response

Returns JSON with base64-encoded WAV audio:

{
  "status": "succeeded",
  "audio": "<base64-encoded WAV>",
  "content_type": "audio/wav",
  "metadata": {
    "duration_s": 12.4,
    "sample_rate": 48000,
    "processing_ms": 8200,
    "seed": 42
  }
}

Architecture

XML prompt (voice + scene + action tags + text)
  -> Gemma 3 12B text encoding
  -> 8-step distilled latent diffusion
  -> Audio VAE decoding
  -> MelBandRoFormer vocal separation (strips SFX unless background_sfx=true)
  -> SeedVC voice identity transfer (when reference provided or multi-chunk)
  -> Output WAV (48kHz stereo)

For longer text, the system splits at sentence boundaries using Kokoro phoneme-level duration estimation and maintains voice continuity between segments via A2V latent conditioning.

VRAM Requirements

VRAM strategy is auto-detected. SageAttention 2 recommended for all configurations.

Performance

Benchmarked on NVIDIA RTX 4090 (24 GB), ~55 seconds of output audio:

Limitations

• Pronunciation: Occasionally garbles complex multi-syllable words and proper nouns.
• 15-second generation window: Each segment capped at ~15s. Longer text splits automatically.
• Emotional range with voice cloning: Identity transfer can reduce emotional extremes. Use a strong archetype in the voice description and provide reference audio with natural emotional variability (10-20 seconds, not monotone).
• Multilingual pronunciation: Language switching mid-speech may cause phonetic drift. Use separate requests per language.
• Generation speed: 3-8 seconds per 15-second segment depending on hardware.
• Reference audio quality: Low-quality references degrade output. Use clean audio with some emotional variability.
• Gemma 3 12B is gated: Requires accepting Google's terms of use and a HuggingFace token with access.

Acknowledgments

• LTX-2 by Lightricks for the base audiovisual model
• Gemma 3 by Google for the text encoder
• SeedVC by Plachta for voice refinement
• Kokoro by hexgrad for duration estimation
• SageAttention for attention acceleration

License

The model weights are released under the LTX-2 Community License Agreement. Scenema Audio's audio diffusion transformer is derived from LTX 2.3's audiovisual model, and its weights are subject to the same terms.

The inference code and server are released under the MIT License.

Gemma 3 12B (text encoder) is a gated model requiring acceptance of Google's terms of use.