moss-va-sft3-emotion-loras

40 emotion adapters for MOSS voice-acting v2 SFT-3

<!-- moss-tied-heads-warning -->

⚠️ Do not merge this adapter into the base weights

merge_and_unload(), merge_adapter(), and any offline "bake the LoRA into the checkpoint" script will destroy the model irrecoverably. This is not a performance caveat. Read this before you write a deployment script.

Why

This adapter targets audio_lm_heads.0 … audio_lm_heads.11 and text_lm_head — 12 of its 23 target modules. In this architecture those output heads are weight-tied to the input embeddings: tie_weights() sets

audio_lm_heads[i].weight  IS  audio_embeddings[i].weight     # the same tensor, not a copy
text_lm_head.weight       IS  transformer.embed_tokens.weight

They are one allocation with two names. So when a merge adds B @ A * (alpha/r) into the head weight, it writes that delta straight into the embedding table at the same time. The model then reads its own inputs through a matrix that has been shifted by an output-side correction. Generation does not fail loudly — it degrades into noise or into a fixed babble, and the damage is inside the checkpoint you just saved. There is nothing to unmerge afterwards, because the original values are gone.

Verify it yourself in three lines

Do not take our word for it:

m = base.model if hasattr(base, "model") else base
print(m.audio_lm_heads[0].weight.data_ptr() == m.audio_embeddings[0].weight.data_ptr())
# True  -> same storage, merging corrupts the embeddings

What to do instead

Load with PEFT and leave the adapter unmerged. Set its strength through the scaling factor:

from peft import PeftModel

model = PeftModel.from_pretrained(base, "<this repo>", adapter_name="a").to(dev).eval()
# do NOT call model.merge_and_unload()

def set_weight(model, name, w):
    """Scale one named adapter's contribution.  alpha/r is its own base scaling."""
    for module in model.modules():
        scaling = getattr(module, "scaling", None)
        if isinstance(scaling, dict) and name in scaling:
            if not hasattr(module, "_base_scaling"):
                module._base_scaling = {}
            module._base_scaling.setdefault(name, scaling[name])
            scaling[name] = module._base_scaling[name] * float(w)

set_weight(model, "a", 1.0)
model.base_model.set_adapter(["a"])      # several adapters can be active at once

This sounds identical to a merge. An unmerged LoRA computes Wx + (B @ A)x * (alpha/r), which is exactly what the merged weight W + B @ A * (alpha/r) would compute — the same arithmetic, in a different order. You give up a small amount of inference speed and you keep the ability to change the weight, stack several adapters, or turn one off. Nothing about the sound changes.

If you are stacking adapters

Set each one's scaling separately and activate them together with model.base_model.set_adapter([...]). Note that stacking is not free: in our own measurements a deep stack held audio quality but destroyed intelligibility (word error 0.063 → 0.554). Add adapters deliberately and measure.

If you maintain code that merges

A regex over module names is not enough — the reliable test is identity of storage. Group the modules by weight.data_ptr() and refuse to merge into any group with more than one member. lora_bank.py in LAION-AI/Humaneness-Voice-Demo-Server does this and asserts on the merge path.

---

One rank-16 LoRA per emotion, each trained on the top 1 % of that emotion across the whole corpus — filtered so that measured naturalness (genuineness) and vocal-burst blend are both above the corpus median.

That filter is the point, not a detail. Measured on a 360-clip listening sample, the median genuineness percentile falls from 0.391 in the top 10 % of an emotion to 0.360 in the top 1 %: the most intense recordings in this corpus are on average the least genuine, so selecting on intensity alone teaches a model to overact.

These adapters are emotion-specific and voice-agnostic — an "Anger" adapter has seen many speakers. For voice identity use the 500 voice adapters instead; the two stack.

What they do, measured

17 adapters compared against the best general checkpoint on identical prompts, identical sampling, identical seed, all prompts from the same three speakers. Each adapter got ten prompts naming its emotion and ten naming no emotion at all.

So they do push their emotion — but they push almost as hard when nobody asked. The selectivity ratio is only about 1.4 : 1, and they cost roughly half again as much transcription error. Do not merge one in permanently and expect neutral speech to survive.

Use the merge weight

A scale sweep over 31 adapters, 8 prompts each, at the extreme band with vocal bursts:

Emotion rises monotonically and, contrary to expectation, genuineness and burst blend rise with it. Timing is untouched at every weight. What breaks is intelligibility, and only between 1.5 and 2.0 — and it breaks as a tail of a few completely derailed clips, not as general decay: the median word error rate is still 0.000 at 1.5 and the mean is at its lowest there.

Recommended operating point: 1.5. Take 2.0 only if you are already generating best-of-N and can discard the failures. For soft or neutral requests, weight 0 — the natural mode is to derive the weight from the requested intensity band, which the prompt already states.

Layout

<Emotion>/adapter_config.json
<Emotion>/adapter_model.safetensors
<Emotion>/bucket.json          # training rows, steps, wall time, prompt-format hash

The 40 adapters

Every adapter: rank 16, alpha 32, 5 epochs, cosine 1e-4 to 5e-6, 34.4 M trainable parameters (0.83 % of the model). All finished with zero non-finite batches.

Inference

import torch, torchaudio
from transformers import AutoProcessor, AutoModel

BASE = "laion/moss-tts-local-transformer-4.55b-voice-acting-v2-sft3"

proc = AutoProcessor.from_pretrained(BASE, trust_remote_code=True)
model = AutoModel.from_pretrained(BASE, trust_remote_code=True,
                                  dtype=torch.bfloat16, attn_implementation="sdpa").cuda().eval()

prompt = open("prompt.txt").read()          # the <user_inst> block, see "How to prompt" below
um = {"role": "user", "content": prompt, "audio_codes_list": []}
b = proc([[um]], mode="generation")

with torch.no_grad():
    out = model.generate(input_ids=b["input_ids"].cuda(),
                         attention_mask=b["attention_mask"].cuda(),
                         max_new_frames=340, do_sample=True,
                         audio_temperature=1.0, audio_top_p=0.95, audio_top_k=50,
                         audio_repetition_penalty=1.0)

# codes -> waveform. Use the processor's own decoder: calling the audio tokenizer directly, or
# reshaping its output, yields a two-channel result that flattens into audio at HALF SPEED and
# still sounds like speech. This project lost a whole corpus to that once.
wav = proc.decode_audio_codes([out_codes], return_stereo=False)[0].reshape(-1).float().cpu()
torchaudio.save("out.flac", wav[None], int(proc.model_config.sampling_rate), format="flac")

Loading adapters

from peft import PeftModel

# one adapter
model = PeftModel.from_pretrained(model, "laion/moss-va-sft3-dpo-lora")

# several, each with its own weight -- the usual case: identity from a voice adapter,
# affect from an emotion adapter, general quality from the DPO adapter.
#
# NOTE: `add_weighted_adapter(..., combination_type="linear")` does NOT work here. It raises
# `ValueError: All adapters must have the same r value`, because the DPO adapter is rank 64 and
# the voice / emotion adapters are rank 16. Activate them together instead and scale each one.
model = PeftModel.from_pretrained(model, "<dpo adapter path>", adapter_name="dpo")
model.load_adapter("<voice adapter path>", adapter_name="voice")
model.load_adapter("<emotion adapter path>", adapter_name="emo")

names = ["dpo", "voice", "emo"]
weights = {"dpo": 1.0, "voice": 1.0, "emo": 1.5}     # 1.5 for emotion is the measured optimum
model.base_model.set_adapter(names)                  # the TUNER takes a list; PeftModel does not
model.active_adapter = names[0]                      # must stay a str or generate() indexes a list

for mod in model.modules():
    sc = getattr(mod, "scaling", None)
    if isinstance(sc, dict):
        if not hasattr(mod, "_base_scaling"):
            mod._base_scaling = dict(sc)
        for k in sc:
            if k in weights:
                sc[k] = mod._base_scaling[k] * weights[k]

Scaling an adapter without re-merging

A LoRA layer computes h + scaling · B(A(x)), so multiplying the stored scaling is the merge weight — exact and reversible:

def set_lora_scale(model, w):
    for mod in model.modules():
        sc = getattr(mod, "scaling", None)
        if isinstance(sc, dict):
            if not hasattr(mod, "_base_scaling"):
                mod._base_scaling = dict(sc)
            for k in sc:
                sc[k] = mod._base_scaling[k] * w

How to prompt this model

Every request is one <user_inst> block. The fields are fixed — none may be added or removed:

<user_inst>
- Reference(s):
{None | Speaker: <name> | <|audio|>}
- Instruction:
{GENERAL: ... and/or SCRIPT: ...}
- Tokens:
{target length in audio frames}
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
{English | German}
- Text:
{the same script as under SCRIPT:, character for character}
</user_inst>

GENERAL: — who is speaking

Prose describing the voice and the clip: age and gender, energy and pace, tension, timbre, clarity, pitch range, breath, affect, which emotions are audible, style, recording quality.

GENERAL: A young adult masculine voice; delivery is normally alert, brisk, neutral tension;
timbre is neutral-toned, fairly smooth; average clarity, wide pitch range, light breath;
affect is mildly positive, slightly dominant; reads as bitterness, contempt; 9.5s, EN.

The phrase reads as … is where the emotion names live.

SCRIPT: — what to say, when, and how

Four kinds of tag, told apart by their brackets:

The disambiguation rule in one line: square bracket = a number of seconds; round bracket with a number = a vocal burst; round bracket without a number = a delivery direction. That is the only thing separating a burst from a direction, which is why directions never carry a number.

A complete example:

<user_inst>
- Reference(s):
None
- Instruction:
GENERAL: A young adult feminine voice, warm and conversational; reads as amusement; 6.0s, EN.
SCRIPT:
[0.4 seconds pause] (intensely amused: letting it out / not hiding it, warm and open,
unguarded; bright, relaxed) [2.4 seconds duration] You are not going to believe this.
[0.3 seconds pause] (breathy giggle, 0.4 seconds) [0.2 seconds pause] (still intensely amused)
[2.3 seconds duration] He actually wore it to the wedding.
- Tokens:
75
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
English
- Text:
[0.4 seconds pause] (intensely amused: letting it out / not hiding it, warm and open,
unguarded; bright, relaxed) [2.4 seconds duration] You are not going to believe this.
[0.3 seconds pause] (breathy giggle, 0.4 seconds) [0.2 seconds pause] (still intensely amused)
[2.3 seconds duration] He actually wore it to the wedding.
</user_inst>

0.4 + 2.4 + 0.3 + 0.4 + 0.2 + 2.3 = 6.0 s = 75 frames. If the numbers do not add up to the token budget the model has to choose which to honour, and length control is the thing it honours best.

Segmentation rules the training data followed

• split at sentence ends (. ! ? …) and at every vocal burst;
• any segment still longer than 12 s is split again at its largest internal gap;
• a duration is measured from the first word onset to the last word offset of that segment, so two sentences of 12 s and 8 s produce [12.0 seconds duration] and [8.0 seconds duration], never a single [20.0 seconds duration];
• gaps below 0.20 s are folded into the neighbouring speech instead of being printed, so the printed numbers still add up;
• a burst that overlaps speech prints only the part that does not overlap.

Vocal-burst labels that actually occur in training

low mumble, ahem, contented sigh, surprised gasp, chuckle, breathy giggle, childlike giggle, wistful sigh, exhausted groan, sharp inhale, resonant hum, scream, yawn, deep breath, soft hum, exasperated sigh, cackle, shriek, coughing, mournful wail, growl, purr.

Realistic lengths: median 0.28 s, 10th percentile 0.14 s, 90th percentile 0.48 s, longest observed 2.46 s. A sigh requested at 3 s is outside anything in the data.

Intensity bands

Delivery directions carry an intensity adverb drawn from the percentile band of the requested emotion. The same cutoffs are used by the training data, the reward and the evaluation:

The family

Independent evaluation

These adapters ship with a measured weight sweep on emotion strength; this is an independent look at what that weight costs elsewhere. The section below is a first measurement, run against the checkpoint they were trained for. It is not a listening test and does not replace one.

Setup. Ten fixed English utterances of two to three sentences, rendered into the SFT3 timed-script format, identical in every condition and in the same order. Base is laion/moss-tts-local-transformer-4.55b-voice-acting-v2-sft3 with laion/moss-va-sft3-dpo-lora-p2 at weight 1.0. One adapter is then added at one weight; nothing else changes, and the sampling seed is fixed. The prompt carries one reference recording of the voice emolia_c1699 from TTS-AGI/moss-voice-profile-references.

Metrics. Word error rate is measured with nvidia/parakeet-tdt-0.6b-v3 against the intended line. Extra words counts words transcribed after the last word that still matches the line — the direct measure of a take running on past its script. Genuineness is laion/voiceclap-commercial-genuineness (0–6) and blend is laion/voiceclap-commercial-vocalburst-blend (0–10). Speaker similarity is ECAPA cosine against that same reference recording.

Baseline, no adapter beyond DPO-p2: word error 0.043, 0.3 extra words, genuineness 1.17, blend 3.35, speaker similarity 0.513.

What this does not tell you. Ten utterances at one seed: the trend across weights is meaningful, a single cell is not. All lines are English, declarative and calm — nothing here describes behaviour on shouting, on German, or on lines carrying vocal bursts. Only one speaker was used. The two scorer models are themselves models, and their absolute levels are not calibrated against human judgement here; differences between conditions are the signal.

Method, all ten prompts, every hyper-parameter and the raw per-take data including transcripts: https://github.com/LAION-AI/Humaneness-Voice-Demo-Server/blob/main/docs/EXPERIMENTS.md

Results

240 conditions: every one of the 40 adapters at six weights.

Averaged over all 40 adapters at each weight

The ten most expensive individual conditions

Reading this. The card's recommended operating point of 1.5 is the most expensive weight measured here on a task that asks only for clean delivery of a written line. The failure mode is not gentle: individual adapters at 1.5 can derail a take completely — Confusion @1.5 reached word error 1.285 with 19 invented words — rather than degrading smoothly.

Downstream this demo moved its default from 1.5 to 1.0, which kept nearly all of the genuineness gain (1.90 against 1.83 in a combined stack) at word error 0.018 instead of 0.083. If you are generating best-of-N and can discard failures, 1.5 remains reasonable; if every take ships, it is not.

<!-- recipes-and-prompting -->

Recipes, prompting and the rest of the stack

These are the emotion adapters — affect without identity. They stack with the voice adapters (identity without affect) and with the talking-style axes.

Where the recipes live

Writing the cue

Brackets carry meaning in this format, and two rules catch nearly everyone out.

Cues are always written in English, even when the spoken line is German. This is how the training corpus is written — German rows read Das zerreißt einen einfach, weißt du? (relief sigh) — so a German cue is out of distribution.

Never write a number inside a bracket. The durations are computed for you and inserted afterwards: [N.N seconds duration] before each speech segment, [N.N seconds pause] for gaps, (label, N.N seconds) for each burst, summed at 12.5 frames per second.

<!-- /recipes-and-prompting -->