LongCat-AudioDiT-Env-TTS-1B-augment

LongCat-AudioDiT Env-TTS — augment (10,000-step fine-tune)

Fine-tune of meituan-longcat/LongCat-AudioDiT-1B for the three-stream env-tts task: given a reference environment audio, a reference speaker audio, and three text streams (env caption / speaker caption / target speech text), generate target speech that places the target text in the referenced environment with the referenced speaker timbre.

This augment variant adds environment-consistent augmentation so the generated target lives in the referenced acoustic scene.

Differences from the base model

Six learnable boundary tokens (three latent-space, three text-space):

latent sequence : [<boe>  z_env  <bos>  z_spk  <bon>  z_target]
text sequence   : [<boe_t> env_text_emb <bos_t> spk_text_emb <bon_t> target_text_emb]

encode_multistream_text(...) is the entry-point; AudioDiTModel.forward(...) also accepts a pre-assembled prompt_latent.

Training summary

Augmentation (the augment change)

Noise + RIR are streamed on-demand from ChristianYang/DNS-Noise (DNS-Challenge noise_fullband + impulse_responses, republished as 24 kHz mono):

• Speaker ref — an independent 50/25/25 draw: clean / noise / noise+RIR, SNR ∈ [−5, 15] dB.
• Env + target (coupled) — a separate 50/25/25 draw whose same noise clip and same RIR are applied to both env and target, placing the generated target in one consistent acoustic scene. The RIR tail is kept; env/target are capped to 15 s.

How to load

Uses custom code in this repo, so pass trust_remote_code=True:

from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained(
    "ChristianYang/LongCat-AudioDiT-Env-TTS-1B-augment",
    trust_remote_code=True,
).cuda().eval()

tokenizer = AutoTokenizer.from_pretrained(model.config.text_encoder_model)

See the training repo's tasks/inference.py for end-to-end env-tts inference.

License

Inherits the original meituan-longcat/LongCat-AudioDiT-1B license.