moss-mediathek-emotion-lora

MOSS Mediathek emotion LoRAs

<!-- moss-tied-heads-warning -->

⚠️ Do not merge this adapter into the base weights

merge_and_unload(), merge_adapter(), and any offline "bake the LoRA into the checkpoint" script will destroy the model irrecoverably. This is not a performance caveat. Read this before you write a deployment script.

Why

This adapter targets audio_lm_heads.0 … audio_lm_heads.11 and text_lm_head — 12 of its 23 target modules. In this architecture those output heads are weight-tied to the input embeddings: tie_weights() sets

audio_lm_heads[i].weight  IS  audio_embeddings[i].weight     # the same tensor, not a copy
text_lm_head.weight       IS  transformer.embed_tokens.weight

They are one allocation with two names. So when a merge adds B @ A * (alpha/r) into the head weight, it writes that delta straight into the embedding table at the same time. The model then reads its own inputs through a matrix that has been shifted by an output-side correction. Generation does not fail loudly — it degrades into noise or into a fixed babble, and the damage is inside the checkpoint you just saved. There is nothing to unmerge afterwards, because the original values are gone.

Verify it yourself in three lines

Do not take our word for it:

m = base.model if hasattr(base, "model") else base
print(m.audio_lm_heads[0].weight.data_ptr() == m.audio_embeddings[0].weight.data_ptr())
# True  -> same storage, merging corrupts the embeddings

What to do instead

Load with PEFT and leave the adapter unmerged. Set its strength through the scaling factor:

from peft import PeftModel

model = PeftModel.from_pretrained(base, "<this repo>", adapter_name="a").to(dev).eval()
# do NOT call model.merge_and_unload()

def set_weight(model, name, w):
    """Scale one named adapter's contribution.  alpha/r is its own base scaling."""
    for module in model.modules():
        scaling = getattr(module, "scaling", None)
        if isinstance(scaling, dict) and name in scaling:
            if not hasattr(module, "_base_scaling"):
                module._base_scaling = {}
            module._base_scaling.setdefault(name, scaling[name])
            scaling[name] = module._base_scaling[name] * float(w)

set_weight(model, "a", 1.0)
model.base_model.set_adapter(["a"])      # several adapters can be active at once

This sounds identical to a merge. An unmerged LoRA computes Wx + (B @ A)x * (alpha/r), which is exactly what the merged weight W + B @ A * (alpha/r) would compute — the same arithmetic, in a different order. You give up a small amount of inference speed and you keep the ability to change the weight, stack several adapters, or turn one off. Nothing about the sound changes.

If you are stacking adapters

Set each one's scaling separately and activate them together with model.base_model.set_adapter([...]). Note that stacking is not free: in our own measurements a deep stack held audio quality but destroyed intelligibility (word error 0.063 → 0.554). Add adapters deliberately and measure.

If you maintain code that merges

A regex over module names is not enough — the reliable test is identity of storage. Group the modules by weight.data_ptr() and refuse to merge into any group with more than one member. lora_bank.py in LAION-AI/Humaneness-Voice-Demo-Server does this and asserts on the merge path.

---

LoRA adapters for laion/moss-tts-local-transformer-4.55b-voice-acting-v2, trained on an emotion-selected subset of German public-broadcast (Mediathek) speech — real recorded human speech, not synthetic.

Nine checkpoints: ranks 16 / 32 / 64 × epochs 1 / 2 / 3.

Read this before using them

On the headline composite score, none of the nine checkpoints is distinguishable from the base model. Measured on a 400-clip held-out set, 40 generations per arm, scored with the VoiceNet/EmoNet/genuineness/blend stack:

No arm reaches ~2 SE against base. diff_vs_base ranges −0.014 to +0.026, |z| ≤ 0.83.

But the composite is hiding a real and consistent trade-off, and that is the actual result:

• Target emotion rises sharply and without exception: 1.468 → 2.17–2.51, i.e. +48 % to +71 % on every single checkpoint. If you want the requested emotion to actually come through, these adapters do that.
• Blend improves everywhere: 4.732 → 5.35–5.82.
• Genuineness falls everywhere: 3.179 → 2.60–2.91. The emotion reads as more performed and less felt.

The composite weights these against each other and they cancel. So the honest summary is not "the training did nothing" — it is "the training buys emotional intensity and burst blending, and pays for it in perceived sincerity." Whether that is a good trade depends entirely on your use case: for expressive character work it usually is; for intimate or documentary-style delivery it usually is not.

Which one to use

r64_e2 is the default: the best composite of the nine, the highest genuineness of any LoRA arm (so the smallest sincerity penalty), and essentially the top target-emotion score. If you want maximum emotional push and can accept lower genuineness, r16_e3 or r64_e3 score marginally higher on target emotion.

Use the usual dose band — 0.35–0.75. Above ~1.0 genuineness and intelligibility collapse.

Usage

import torch
from transformers import AutoModel, AutoProcessor
from peft import PeftModel

REPO = "laion/moss-tts-local-transformer-4.55b-voice-acting-v2"
proc = AutoProcessor.from_pretrained(
    REPO, trust_remote_code=True,
    codec_path="OpenMOSS-Team/MOSS-Audio-Tokenizer-v2")
model = AutoModel.from_pretrained(          # AutoModel, NOT AutoModelForCausalLM
    REPO, trust_remote_code=True,
    dtype=torch.bfloat16, attn_implementation="sdpa").cuda().eval()

model = PeftModel.from_pretrained(model, "laion/moss-mediathek-emotion-lora/r64_e2").eval()

conv = [[proc.build_user_message(
    text="Ich habe wirklich alles versucht.",           # spoken words ONLY
    instruction='GENERAL: A tired middle-aged man, close mic.\n'
                'SCRIPT:\n(quietly, on the edge of giving up) "Ich habe wirklich alles versucht."',
    language="German",
    tokens=6)]]
b = proc(conv, mode="generation")
out = model.generate(input_ids=b["input_ids"].cuda(),
                     attention_mask=b["attention_mask"].cuda(),
                     max_new_frames=400, do_sample=True,
                     audio_temperature=1.0, audio_top_p=0.95, audio_top_k=30)
wav = proc.decode(out)[0].audio_codes_list[0].cpu().float().numpy()   # 48 kHz

Notes that will otherwise cost you an afternoon:

• Load the base with AutoModel, not AutoModelForCausalLM.
• Missing audio_lm_heads.* / text_lm_head.weight warnings on load are benign.
• instruction holds the whole GENERAL: …\nSCRIPT:… caption; text holds only the spoken words. Empty fields render as the literal string "None".

Training

A methodological note

Checkpoints here are ranked by generating audio and scoring it, not by validation loss. On a sibling run, validation loss rose from 4.42 to 5.44 across epochs 1→8 — which reads as severe overfitting — while listeners scored the two checkpoints identically. Validation loss has now disagreed with listening three times on this model family. Do not select checkpoints with it.

Links

• Manual: https://laion-ai.github.io/moss-voiceacting-manual/
• Pipeline notes: https://github.com/LAION-AI/Voice-Acting-Pipeline-WIP
• Emotion LoRAs (LAION's Got Talent): https://huggingface.co/TTS-AGI/moss-emotion-loras-v3
• Vocal-burst LoRAs: https://huggingface.co/laion/vocal-burst-lora-adapters