moss-va-sft3-quality-lora-adapters

Quality LoRA adapters for MOSS voice-acting SFT3 — genuineness, vocal-burst blend, aesthetics

<!-- moss-tied-heads-warning -->

⚠️ Do not merge this adapter into the base weights

merge_and_unload(), merge_adapter(), and any offline "bake the LoRA into the checkpoint" script will destroy the model irrecoverably. This is not a performance caveat. Read this before you write a deployment script.

Why

This adapter targets audio_lm_heads.0 … audio_lm_heads.11 and text_lm_head — 12 of its 23 target modules. In this architecture those output heads are weight-tied to the input embeddings: tie_weights() sets

audio_lm_heads[i].weight  IS  audio_embeddings[i].weight     # the same tensor, not a copy
text_lm_head.weight       IS  transformer.embed_tokens.weight

They are one allocation with two names. So when a merge adds B @ A * (alpha/r) into the head weight, it writes that delta straight into the embedding table at the same time. The model then reads its own inputs through a matrix that has been shifted by an output-side correction. Generation does not fail loudly — it degrades into noise or into a fixed babble, and the damage is inside the checkpoint you just saved. There is nothing to unmerge afterwards, because the original values are gone.

Verify it yourself in three lines

Do not take our word for it:

m = base.model if hasattr(base, "model") else base
print(m.audio_lm_heads[0].weight.data_ptr() == m.audio_embeddings[0].weight.data_ptr())
# True  -> same storage, merging corrupts the embeddings

What to do instead

Load with PEFT and leave the adapter unmerged. Set its strength through the scaling factor:

from peft import PeftModel

model = PeftModel.from_pretrained(base, "<this repo>", adapter_name="a").to(dev).eval()
# do NOT call model.merge_and_unload()

def set_weight(model, name, w):
    """Scale one named adapter's contribution.  alpha/r is its own base scaling."""
    for module in model.modules():
        scaling = getattr(module, "scaling", None)
        if isinstance(scaling, dict) and name in scaling:
            if not hasattr(module, "_base_scaling"):
                module._base_scaling = {}
            module._base_scaling.setdefault(name, scaling[name])
            scaling[name] = module._base_scaling[name] * float(w)

set_weight(model, "a", 1.0)
model.base_model.set_adapter(["a"])      # several adapters can be active at once

This sounds identical to a merge. An unmerged LoRA computes Wx + (B @ A)x * (alpha/r), which is exactly what the merged weight W + B @ A * (alpha/r) would compute — the same arithmetic, in a different order. You give up a small amount of inference speed and you keep the ability to change the weight, stack several adapters, or turn one off. Nothing about the sound changes.

If you are stacking adapters

Set each one's scaling separately and activate them together with model.base_model.set_adapter([...]). Note that stacking is not free: in our own measurements a deep stack held audio quality but destroyed intelligibility (word error 0.063 → 0.554). Add adapters deliberately and measure.

If you maintain code that merges

A regex over module names is not enough — the reliable test is identity of storage. Group the modules by weight.data_ptr() and refuse to merge into any group with more than one member. lora_bank.py in LAION-AI/Humaneness-Voice-Demo-Server does this and asserts on the merge path.

---

Three rank-16 adapters, one per perceptual quality axis, each trained on the top 1 % of a 3,144,739-utterance annotated corpus along that one axis, balanced over age and gender. For laion/moss-tts-local-transformer-4.55b-voice-acting-v2-sft3.

⚠️ Read before loading

Do not merge. They target audio_lm_heads.0…11, and audio_lm_heads.N.weight is audio_embeddings.N.weight — weight-tied. A merge writes the delta into the audio embedding table too; measured, both tensors moved by exactly 6.103515625e-05. Load as PEFT adapters.

Unevaluated. All three trained cleanly, no non-finite loss or gradient in 45,000 steps. No listening test, no automatic scoring, no A/B against the bare base has been run.

Usage

import torch
from transformers import AutoModel, AutoProcessor
from peft import PeftModel

BASE = "laion/moss-tts-local-transformer-4.55b-voice-acting-v2-sft3"
REPO = "laion/moss-va-sft3-quality-lora-adapters"

proc  = AutoProcessor.from_pretrained(BASE, trust_remote_code=True)
model = AutoModel.from_pretrained(BASE, trust_remote_code=True,
                                  dtype=torch.bfloat16, attn_implementation="sdpa").cuda()
model = PeftModel.from_pretrained(model, REPO, subfolder="adapters/genuineness_high",
                                  adapter_name="genuineness")
model.eval()

Scale the effect by multiplying PEFT's stored scaling (alpha/r = 2.0); w = 0 is the bare base, w = 1 the adapter as trained:

def set_lora_scale(model, w, adapter="genuineness"):
    for m in model.modules():
        if hasattr(m, "scaling") and adapter in getattr(m, "scaling", {}):
            if not hasattr(m, "_base_scaling"):
                m._base_scaling = dict(m.scaling)
            m.scaling[adapter] = m._base_scaling[adapter] * w

Prompting

Trained with the v3 prompt library, PROMPT_FORMAT_HASH = 073aeb09dc923376. Prompt exactly as you prompt the base model:

<user_inst>
- Reference(s):
{reference}
- Instruction:
{instruction}
- Tokens:
{tokens}
- Quality:
{quality}
- Sound Event:
{sound_event}
- Ambient Sound:
{ambient_sound}
- Language:
{language}
- Text:
{text}
</user_inst>

Instruction carries GENERAL: (one sentence describing voice and performance, ending with genuineness x/6; vocal-burst blend x/10; <duration>s, <LANG>.) and SCRIPT: (the timed script — [D seconds duration] per sentence, [G seconds pause] for gaps ≥ 0.20 s, (label, D seconds) for a vocal burst, and (direction) without a number for delivery). Tokens is round(seconds × 12.5); the tokenizer runs at 12.5 fps.

Selection

The corpus is 3,144,739 annotated utterances. For each axis, the top 1 % on that axis was cut, then balanced over age × gender from the row's own VoiceNet vector — age at the corpus tertiles, gender at the median — filling cells evenly and topping up from the largest.

• genuineness_high — gated on blend ≥ median. 23,279 candidates → 12,000 balanced.
• blend_high — gated on genuineness ≥ median and on the clip actually carrying a detected burst: a blend score on a clip with no burst is a score for something that is not there. Balanced over age × gender × dominant burst class, because low mumble and ahem are 45 % and 31 % of all detections and would otherwise be the whole adapter. 12,588 → 12,000.
• esthetics_high — no cross-gate, and that is a measurement, not an oversight: the top 1 % of ESTH intersected with "blend and genuineness both above median" is 364 rows out of 31,447. The aesthetic tail of this corpus is almost disjoint from the other two quality axes, so gating a quality target on two other quality axes would select their agreement rather than aesthetics. The target is its own gate. 31,453 → 12,000.

One observation from training: esthetics_high sat at a markedly lower loss than the other two (3.80 against 5.04 at the same step) — the aesthetic tail is the easier distribution to model.

Training

Related

Limitations

1. Unevaluated, as above.
2. The targets are model outputs. "The top 1 % of genuineness" means the top 1 % as that scorer ranks it, including its errors and speaker priors.
3. genuineness is the one attribute with measurable speaker memorisation in this corpus: a probe on the model's activations scores +0.042 higher when speakers are allowed to leak across the split, against ≈0 for emotion and voice quality. Some of what this adapter learned may be speaker identity rather than authenticity.
4. Aesthetics is nearly orthogonal to the other two axes here, as the 364-row intersection shows. Stacking esthetics_high with genuineness_high has not been tested.

Trained on JUPITER (Jülich Supercomputing Centre, project reformo), GH200, August 2026.

Independent evaluation

These adapters shipped marked unevaluated. The section below is a first measurement, run against the checkpoint they were trained for. It is not a listening test and does not replace one.

Setup. Ten fixed English utterances of two to three sentences, rendered into the SFT3 timed-script format, identical in every condition and in the same order. Base is laion/moss-tts-local-transformer-4.55b-voice-acting-v2-sft3 with laion/moss-va-sft3-dpo-lora-p2 at weight 1.0. One adapter is then added at one weight; nothing else changes, and the sampling seed is fixed. The prompt carries one reference recording of the voice emolia_c1699 from TTS-AGI/moss-voice-profile-references.

Metrics. Word error rate is measured with nvidia/parakeet-tdt-0.6b-v3 against the intended line. Extra words counts words transcribed after the last word that still matches the line — the direct measure of a take running on past its script. Genuineness is laion/voiceclap-commercial-genuineness (0–6) and blend is laion/voiceclap-commercial-vocalburst-blend (0–10). Speaker similarity is ECAPA cosine against that same reference recording.

Baseline, no adapter beyond DPO-p2: word error 0.043, 0.3 extra words, genuineness 1.17, blend 3.35, speaker similarity 0.513.

What this does not tell you. Ten utterances at one seed: the trend across weights is meaningful, a single cell is not. All lines are English, declarative and calm — nothing here describes behaviour on shouting, on German, or on lines carrying vocal bursts. Only one speaker was used. The two scorer models are themselves models, and their absolute levels are not calibrated against human judgement here; differences between conditions are the signal.

Method, all ten prompts, every hyper-parameter and the raw per-take data including transcripts: https://github.com/LAION-AI/Humaneness-Voice-Demo-Server/blob/main/docs/EXPERIMENTS.md

Results

Genuineness. Raises its own measured score at low weight — 1.57 at 0.25 against a 1.17 baseline — and stops helping above that. Above 1.0 it breaks intelligibility: word error 0.176 at 1.25 and 0.177 at 1.5, with invented words in 60 % of takes at 1.5, while the blend score falls from 3.26 to 1.59. Useful range is roughly 0.25–0.5.

Blend. The safest of the three. Word error stays between 0.013 and 0.061 across the whole range with no upward trend, and the blend score rises above baseline. At 0.25 it also produced the highest genuineness score of the entire study (1.92).

Aesthetics. Cheap on intelligibility at every weight measured, but it costs genuineness monotonically: 1.64 at 0.25 down to 0.79 at 1.25 and 1.5. It and the genuineness adapter pull in opposite directions.

Adopted downstream. The demo these measurements come from now runs genuineness at 0.25, blend at 0.5 and aesthetics at 0.5, rather than all three at 1.0. On the same ten utterances that moved word error from 0.116 to 0.055.

<!-- recipes-and-prompting -->

Recipes, prompting and the rest of the stack

These are the quality and aesthetics adapters — one per perceptual axis. They are the ones you leave on by default rather than the ones you reach for per line.

Where the recipes live

Writing the cue

Brackets carry meaning in this format, and two rules catch nearly everyone out.

Cues are always written in English, even when the spoken line is German. This is how the training corpus is written — German rows read Das zerreißt einen einfach, weißt du? (relief sigh) — so a German cue is out of distribution.

Never write a number inside a bracket. The durations are computed for you and inserted afterwards: [N.N seconds duration] before each speech segment, [N.N seconds pause] for gaps, (label, N.N seconds) for each burst, summed at 12.5 frames per second.

<!-- /recipes-and-prompting -->