Downloads · 30 days
13
9% of all-time downloads
laion/moss-tts-local-transformer-4.55b-voice-acting-v2-sft
moss-tts-local-transformer-4.55b-voice-acting-v2-sft is a text-to-speech model from laion. Use it when you need text read aloud. It is set up for transformers. The card lists the license as cc-by-4.0.
A full fine-tune of laion/moss-tts-local-transformer-4.55b-voice-acting-v2 on 3,147,802 annotated utterances — half synthetic voice profiles, half real speech in English and German — for three epochs.
Downloads · 30 days
13
9% of all-time downloads
All-time downloads
142
Public
Parameters
4.1B
8.3 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors8.3 GB · 100%
From the Hugging Face model README
A full fine-tune of
laion/moss-tts-local-transformer-4.55b-voice-acting-v2
on 3,147,802 annotated utterances — half synthetic voice profiles, half real speech in English
and German — for three epochs.
By Christoph Schuhmann and LAION.
Held-out validation loss fell at every one of the twelve evaluations, on both constituent datasets independently: 4.7076 → 4.6314.
This is not an adapter. All 4.13 B parameters were trained.
| samples seen | 9,443,406 sample-passes (3,147,802 × 3 epochs) |
| peak learning rate | 7e-6, cosine, warmup over the first half of epoch 1, decayed to 10 % |
| optimizer | AdamW, β (0.9, 0.95), ε 1e-8, weight decay 0.1, grad-norm clip 1.0 |
| precision | fp32 master weights, bf16 compute |
| global batch | 4,096 samples |
| steps | 2,232 |
| hardware | 64 nodes × 4 GH200 = 256 GPUs, 1 h 53 min |
| loss | channel-wise cross-entropy, 1 : 32 weighting of the stop channel against the 12 audio codebooks |
Learning rate, and why 7e-6. The reference trainer for this model family defaults to 1e-5, but that default is for a small fine-tune at an effective batch of ~32. Here an already-specialised checkpoint is fully fine-tuned over 9.4 M sample-passes at a global batch of 4,096. A long full-fine-tune schedule on a good checkpoint argues for a lower rate; a large batch (less gradient noise) argues for a higher one. 7e-6 is 0.7× the reference default.
Held out by blake2b(uid) % 1000 == 0 — deterministic, disjoint from training, and covering both
datasets. Loss is reported per dataset so a regression on one cannot hide behind a gain on the
other.
| step | val loss | voice profiles | real speech |
|---|---|---|---|
| 186 | 4.7076 | 4.4786 | 4.8668 |
| 558 | 4.6621 | 4.4342 | 4.8206 |
| 930 | 4.6472 | 4.4199 | 4.8053 |
| 1302 | 4.6370 | 4.4090 | 4.7956 |
| 1674 | 4.6328 | 4.4047 | 4.7915 |
| 2232 | 4.6314 | 4.4030 | 4.7902 |
Monotone throughout, no divergence between the two datasets, no sign of overfitting at three epochs. Voice profiles sit ~0.39 nats below real speech, which is expected — synthetic single-voice material is more predictable than recordings made in the wild.
Listen before you trust the number: the validation sample grid pairs the base model, this fine-tune and the ground truth on the same held-out prompts.
The model expects the MOSS <user_inst> block. The exact prompt surface it was trained on
matters — the training mix deliberately varied it, and departing from these conventions will
cost you quality.
<user_inst>
- Reference(s):
{<|audio|> | Speaker: <name> | None}
- Instruction:
{caption}
- Tokens:
{target frame count, 12.5 fps}
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
{English | German}
- Text:
{[7.3 seconds duration] }{transcript}
</user_inst>
What the training mix contained, and in what proportion:
| element | training distribution |
|---|---|
| Reference slot | reference audio 50 % / Speaker: <name> 50 % for voice profiles; reference-or-None for real speech |
| Instruction | four surfaces: one of 16 rendered caption templates (30 %), GENERAL: …\nSCRIPT:\n… (45 %), GENERAL: only (15 %), SCRIPT: only (10 %) |
| Duration tag | [7.3 seconds duration] prefixed to the text on 50 % of samples |
| Inline burst cues | parenthesised cues such as (laughs) present, but stripped entirely from both text and caption on 10 % of samples |
The duration tag is spoken duration — last word offset minus first word onset — not clip duration. Clip duration overstates it by ~1.05 s on voice profiles and ~0.40 s on real speech because of leading and trailing silence. If you tag with clip duration the model will pad.
Reference audio is passed as codec codes, [T, 12] int64, directly in audio_codes_list; no audio
tokenizer is needed at inference for conditioning.
import torch
from transformers import AutoProcessor, AutoModel
REPO = "laion/moss-tts-local-transformer-4.55b-voice-acting-v2-sft"
proc = AutoProcessor.from_pretrained(REPO, trust_remote_code=True,
codec_path="OpenMOSS-Team/MOSS-Audio-Tokenizer-v2")
model = AutoModel.from_pretrained(REPO, trust_remote_code=True,
dtype=torch.bfloat16, attn_implementation="sdpa").cuda().eval()
msg = proc.build_user_message(
text="[4.2 seconds duration] I really did not see that coming.",
instruction="GENERAL: A young adult feminine voice; delivery is bright and quick; "
"affect is positive, animated.\nSCRIPT:\n\"I really did not see that coming.\"",
tokens=53, language="English")
batch = proc([[msg]], mode="generation")
out = model.generate(input_ids=batch["input_ids"].cuda(),
attention_mask=batch["attention_mask"].cuda(),
max_new_frames=400, do_sample=True,
audio_temperature=1.0, audio_top_p=0.95, audio_top_k=50)
wav = proc.decode(out)[0].audio_codes_list[0] # 48 kHz
tokens is the target length in codec frames at 12.5 fps — 4.2 s ≈ 53 frames.
| dataset | rows | share of each batch |
|---|---|---|
laion/laion-voice-profiles-sft | 1,200,531 | 50 % |
laion/tts-realspeech-sft-en-de | 1,947,272 | 50 % |
Every micro-batch was exactly half of each dataset — the sampler alternates strictly between the two pools rather than mixing in expectation. Verified on 20 real batches: 0.5000 / 0.5000 on every one.
Because the two datasets are unequal in size and each batch is balanced, over three nominal epochs the voice-profile rows are seen ≈3.93 times and the real-speech rows ≈2.42 times.
The voice-profile source ships strictly one voice per file, so the corpus was globally shuffled before training; each shard ends up containing all 500 voices. Measured on real batches: 7.75 distinct voices per 8-sample batch, and no single voice ever exceeded 25 % of a batch.
Research on expressive and controllable speech synthesis. Not validated for, and should not be used for, generating speech attributed to a real person without their consent, or for any application where a synthetic voice could be mistaken for a specific individual.
Released under CC-BY-4.0 by Christoph Schuhmann and LAION.
The base model laion/moss-tts-local-transformer-4.55b-voice-acting-v2 is Apache-2.0, and that
notice is preserved: this work is a derivative of it. The MOSS-TTS architecture and the
MOSS-Audio-Tokenizer-v2 codec originate with the OpenMOSS team.
@misc{schuhmann2026mossva_sft,
title = {MOSS Voice-Acting 4.55B --- SFT},
author = {Schuhmann, Christoph and {LAION}},
year = {2026},
url = {https://huggingface.co/laion/moss-tts-local-transformer-4.55b-voice-acting-v2-sft},
note = {Full fine-tune on 3.15M annotated utterances; CC-BY-4.0}
}