moss-mediathek-hq-lora

MOSS Voice-Acting — German Mediathek HQ LoRAs

<!-- moss-tied-heads-warning -->

⚠️ Do not merge this adapter into the base weights

merge_and_unload(), merge_adapter(), and any offline "bake the LoRA into the checkpoint" script will destroy the model irrecoverably. This is not a performance caveat. Read this before you write a deployment script.

Why

This adapter targets audio_lm_heads.0 … audio_lm_heads.11 and text_lm_head — 12 of its 23 target modules. In this architecture those output heads are weight-tied to the input embeddings: tie_weights() sets

audio_lm_heads[i].weight  IS  audio_embeddings[i].weight     # the same tensor, not a copy
text_lm_head.weight       IS  transformer.embed_tokens.weight

They are one allocation with two names. So when a merge adds B @ A * (alpha/r) into the head weight, it writes that delta straight into the embedding table at the same time. The model then reads its own inputs through a matrix that has been shifted by an output-side correction. Generation does not fail loudly — it degrades into noise or into a fixed babble, and the damage is inside the checkpoint you just saved. There is nothing to unmerge afterwards, because the original values are gone.

Verify it yourself in three lines

Do not take our word for it:

m = base.model if hasattr(base, "model") else base
print(m.audio_lm_heads[0].weight.data_ptr() == m.audio_embeddings[0].weight.data_ptr())
# True  -> same storage, merging corrupts the embeddings

What to do instead

Load with PEFT and leave the adapter unmerged. Set its strength through the scaling factor:

from peft import PeftModel

model = PeftModel.from_pretrained(base, "<this repo>", adapter_name="a").to(dev).eval()
# do NOT call model.merge_and_unload()

def set_weight(model, name, w):
    """Scale one named adapter's contribution.  alpha/r is its own base scaling."""
    for module in model.modules():
        scaling = getattr(module, "scaling", None)
        if isinstance(scaling, dict) and name in scaling:
            if not hasattr(module, "_base_scaling"):
                module._base_scaling = {}
            module._base_scaling.setdefault(name, scaling[name])
            scaling[name] = module._base_scaling[name] * float(w)

set_weight(model, "a", 1.0)
model.base_model.set_adapter(["a"])      # several adapters can be active at once

This sounds identical to a merge. An unmerged LoRA computes Wx + (B @ A)x * (alpha/r), which is exactly what the merged weight W + B @ A * (alpha/r) would compute — the same arithmetic, in a different order. You give up a small amount of inference speed and you keep the ability to change the weight, stack several adapters, or turn one off. Nothing about the sound changes.

If you are stacking adapters

Set each one's scaling separately and activate them together with model.base_model.set_adapter([...]). Note that stacking is not free: in our own measurements a deep stack held audio quality but destroyed intelligibility (word error 0.063 → 0.554). Add adapters deliberately and measure.

If you maintain code that merges

A regex over module names is not enough — the reliable test is identity of storage. Group the modules by weight.data_ptr() and refuse to merge into any group with more than one member. lora_bank.py in LAION-AI/Humaneness-Voice-Demo-Server does this and asserts on the merge path.

---

Three PEFT/LoRA adapters for laion/moss-tts-local-transformer-4.55b-voice-acting-v2, trained on a high-quality expressive subset of German public-broadcast speech — 43,612 segments, 185 hours, selected from a 4.9-million-segment parent corpus.

🎧 Listen — 7 emotions × German/English, with and without the emotion adapter

---

Which one to take

r64_epoch3 is the default: it has the lowest validation loss and the most capacity for a corpus this size. Drop to r32 or r16 if you are stacking several adapters or care about download size — the three are within 0.02 of each other on validation loss, which on this stack is not a meaningful gap.

Full per-epoch curves are in train_history.json.

⚠️ Validation loss has repeatedly failed to rank checkpoints on this stack — four separate times now, including a case where a 0.905 loss regression was inaudible. The table above is reported because it is what was measured, not because it is a reliable quality ordering. Listen to the demo grid before choosing.

---

What it does, and the one thing to know first

The adapter pulls the base model toward real German broadcast delivery — the register of public-service documentary, reportage and interview audio, which is where the training data comes from.

⚠️ It is a German adapter. English output runs long.

Measured on the demo grid: German lands at 3.7–4.8 s for a 9-word line (~2.4 words/s, natural for the language). English reaches 10–22 s for a 13-word line — far past natural pacing; the model stretches and pads rather than speaking. Stacking an emotion adapter makes English worse (joy 10.0 → 22.3 s, sadness 8.2 → 17.6 s) and barely moves German.

Use it for German. If you use it for English, judge it on register and expect to control length explicitly with tokens and max_new_frames.

---

Quickstart

import torch, soundfile as sf
from transformers import AutoProcessor, AutoModel
from peft import PeftModel

BASE  = "laion/moss-tts-local-transformer-4.55b-voice-acting-v2"
CODEC = "OpenMOSS-Team/MOSS-Audio-Tokenizer-v2"

# AutoModel, NOT AutoModelForCausalLM -- MossTTSLocalConfig is not registered for the
# CausalLM auto-class and from_pretrained raises "Unrecognized configuration class".
proc  = AutoProcessor.from_pretrained(BASE, trust_remote_code=True, codec_path=CODEC)
model = AutoModel.from_pretrained(
    BASE, trust_remote_code=True, dtype=torch.bfloat16,
    attn_implementation="sdpa",          # flash-attn 2.x is incompatible with this model
).cuda().eval()

pm = PeftModel.from_pretrained(
    model, "laion/moss-mediathek-hq-lora", subfolder="r64_epoch3", adapter_name="MTH"
).eval()

# `instruction` is the whole director's note; `text` is ONLY the spoken words.
# Empty fields render as the literal string "None".
instruction = ("GENERAL: A natural adult voice, clean studio capture, genuine unperformed "
               "delivery; clearly sad, heavy and slowed, the voice thickening.\n"
               'SCRIPT:\n(traurig) "Ich hatte alles genau geplant" (quiet sob) '
               '"und dann kam dieser Anruf."')
text = "Ich hatte alles genau geplant und dann kam dieser Anruf."

conv  = [[proc.build_user_message(text=text, instruction=instruction, language="German",
                                  tokens=int(len(text.split()) / 2.78 * 12.5))]]
batch = proc(conv, mode="generation")

with torch.no_grad():
    out = pm.generate(input_ids=batch["input_ids"].cuda(),
                      attention_mask=batch["attention_mask"].cuda(),
                      max_new_frames=320, do_sample=True,
                      text_temperature=0.7, text_top_k=50, text_top_p=1.0,
                      audio_temperature=1.0, audio_top_k=30, audio_top_p=0.95,
                      audio_repetition_penalty=1.1)

msg = proc.decode(out)[0]
w = msg.audio_codes_list[0].cpu().float().numpy()   # ALREADY a waveform -- do not decode again
if w.ndim > 1:
    w = w.mean(0)
sf.write("out.wav", w, 48000)

audio_lm_heads.* / text_lm_head.weight reported MISSING at load is benign — those heads are weight-tied.

Stacking with emotion and vocal-burst adapters

pm.load_adapter("TTS-AGI/moss-emotion-loras-v3", subfolder="Sadness", adapter_name="Sadness")
pm.load_adapter("laion/vocal-burst-lora-adapters", subfolder="quiet_sob", adapter_name="sob")
pm.base_model.set_adapter(["MTH", "Sadness", "sob"])

Doses used in the demo grid, following the manual: Mediathek 1.0 · emotion 0.5 · burst 0.5. The burst dose matters — 0.75–1.0 raises burst probability but eats the words after the burst (tail coverage 0.90 at λ=0.5 vs 0.45 at λ=1.0).

See the manual for the set_dose helper if you want per-adapter merge control.

---

Training data

A two-half high-quality subset of the German public-broadcast Mediathek corpus:

Reports with the full per-class breakdown are in selection_half1_report.json and selection_half2_report.json. The dataset itself is at TTS-AGI/german-mediathek-hq-expressive (private).

Training: rank 16/32/64 trained in one run against a shared frozen bf16 base (so the rank comparison is paired), alpha = 2 × rank, lora_dropout = 0.05, lr 2e-4 linear decay, 3 epochs, 6,903 optimiser steps, 7 h 34 m on one GH200. Targets are the global q/k/v/o/gate/up/down projections, the local decoder's c_attn/c_proj/fc_in/fc_out, and all 12 audio_lm_heads — the audio heads matter; adapting attention alone moves the voice much less.

---

Where everything lives

Caveats

• German adapter. English works but runs long — see the box above.
• No human listening evaluation was run on these adapters; the demo grid is provided so you can make that judgement yourself. Validation loss is reported but has a poor track record here.
• Trained on public-broadcast material; the register it pulls toward is documentary/reportage, not drama.
• Scores quoted in the demo grid come from model-based evaluators, not human raters.

Provenance

Trained by LAION as part of the MOSS voice-acting line. Full experimental record: LAION-AI/laion-moss-local-1.5-voice-acting-4.55b.