scena

ScenA — Reference-Driven Multi-Speaker Audio Scene Generation

ScenA generates multi-speaker audio scenes (dialogue, conversation) from a text prompt, conditioned on one or more reference-audio clips that set the speakers' voices. It is an audio-only, reference-conditioned text-to-audio flow-matching DiT built on the LTX-2 architecture (~4B parameters, 48 layers).

• 📄 Paper: arXiv:2606.19325
• 🌐 Project page: https://finmickey.github.io/scena/
• 💻 Code: https://github.com/finmickey/scena
• 🏢 Lightricks · Tel Aviv University

Files

You also need the Gemma-3-12B-IT text encoder (google/gemma-3-12b-it, gated on HF).

Quickstart

git clone https://github.com/finmickey/scena
cd scena
pip install -e .   # or: uv sync

# download the checkpoint + audio VAE bundle
huggingface-cli download mifinkelson/scena scena.safetensors audio_vae.safetensors --local-dir ./checkpoints

Two example reference voices ship with the repo under examples/references/ — the same two voices drive both examples below, so this runs out of the box:

from ltx_pipelines.t2aud_ref_cond import T2AudRefCondPipeline

pipe = T2AudRefCondPipeline(
    checkpoint_path="checkpoints/scena.safetensors",
    audio_vae_path="checkpoints/audio_vae.safetensors",
    gemma_root="/path/to/google/gemma-3-12b-it",
)

refs = ["examples/references/reference_1.wav", "examples/references/reference_2.wav"]

# A simple two-speaker dialogue
pipe(
    prompt='The speaker from reference 1 says: "The taxi drivers are on strike again." The speaker from reference 2 says: "What for?" The speaker from reference 1 says: "They want the government to reduce the price of the gasoline." The speaker from reference 2 says: "It is really a hot potato."',
    ref_audio_paths=refs,
    duration=7.0,
    seed=1,
).save("dialogue.wav")

# Farm at sunrise — the same two voices in a scene with sound effects
pipe(
    prompt='A farm at sunrise: a rooster crows. Chickens cluck softly throughout. The speaker from reference 1 says with a yawn: "Way too early for this." The speaker from reference 2 chuckles: "Welcome to country life." The rooster crows again.',
    ref_audio_paths=refs,
    duration=8.0,
    seed=1,
).save("farm.wav")

Prompting guide

• Refer to speakers as "the speaker from reference 1", "reference 2", … matching the order of ref_audio_paths.
• Put the spoken words in quotes; describe sound effects / ambience in plain prose (e.g. "rain drums steadily on a tin roof", "a roaring stadium crowd cheers").
• Reference clips: clean single-speaker speech, up to ~20 s each — the more the better (any sample rate; mono or stereo).
• You don't have to follow a strict turn-taking pattern; see the demo page for more varied examples.
• 60 inference steps, guidance ~7, and output durations up to ~20 s work best (the model was trained on ≤20 s scenes).

Citation

@article{finkelson2026scena,
  title   = {Reference-Driven Multi-Speaker Audio Scene Generation from In-the-Wild Priors},
  author  = {Finkelson, Michael and Segal, Daniel and Richardson, Eitan and Armon, Shahar and Goldring, Nani and Panet, Poriya and Zabari, Nir and Brazowski, Benjamin and Patashnik, Or and HaCohen, Yoav},
  journal = {arXiv preprint arXiv:2606.19325},
  year    = {2026}
}

License

Released under the LTX-2 Community License.