moss-va-sft3-vocal-burst-lora-adapters

Vocal-burst LoRA adapters for MOSS voice-acting SFT3 — 70 classes + 1 general, rank 16

<!-- moss-tied-heads-warning -->

⚠️ Do not merge this adapter into the base weights

merge_and_unload(), merge_adapter(), and any offline "bake the LoRA into the checkpoint" script will destroy the model irrecoverably. This is not a performance caveat. Read this before you write a deployment script.

Why

This adapter targets audio_lm_heads.0 … audio_lm_heads.11 and text_lm_head — 12 of its 23 target modules. In this architecture those output heads are weight-tied to the input embeddings: tie_weights() sets

audio_lm_heads[i].weight  IS  audio_embeddings[i].weight     # the same tensor, not a copy
text_lm_head.weight       IS  transformer.embed_tokens.weight

They are one allocation with two names. So when a merge adds B @ A * (alpha/r) into the head weight, it writes that delta straight into the embedding table at the same time. The model then reads its own inputs through a matrix that has been shifted by an output-side correction. Generation does not fail loudly — it degrades into noise or into a fixed babble, and the damage is inside the checkpoint you just saved. There is nothing to unmerge afterwards, because the original values are gone.

Verify it yourself in three lines

Do not take our word for it:

m = base.model if hasattr(base, "model") else base
print(m.audio_lm_heads[0].weight.data_ptr() == m.audio_embeddings[0].weight.data_ptr())
# True  -> same storage, merging corrupts the embeddings

What to do instead

Load with PEFT and leave the adapter unmerged. Set its strength through the scaling factor:

from peft import PeftModel

model = PeftModel.from_pretrained(base, "<this repo>", adapter_name="a").to(dev).eval()
# do NOT call model.merge_and_unload()

def set_weight(model, name, w):
    """Scale one named adapter's contribution.  alpha/r is its own base scaling."""
    for module in model.modules():
        scaling = getattr(module, "scaling", None)
        if isinstance(scaling, dict) and name in scaling:
            if not hasattr(module, "_base_scaling"):
                module._base_scaling = {}
            module._base_scaling.setdefault(name, scaling[name])
            scaling[name] = module._base_scaling[name] * float(w)

set_weight(model, "a", 1.0)
model.base_model.set_adapter(["a"])      # several adapters can be active at once

This sounds identical to a merge. An unmerged LoRA computes Wx + (B @ A)x * (alpha/r), which is exactly what the merged weight W + B @ A * (alpha/r) would compute — the same arithmetic, in a different order. You give up a small amount of inference speed and you keep the ability to change the weight, stack several adapters, or turn one off. Nothing about the sound changes.

If you are stacking adapters

Set each one's scaling separately and activate them together with model.base_model.set_adapter([...]). Note that stacking is not free: in our own measurements a deep stack held audio quality but destroyed intelligibility (word error 0.063 → 0.554). Add adapters deliberately and measure.

If you maintain code that merges

A regex over module names is not enough — the reliable test is identity of storage. Group the modules by weight.data_ptr() and refuse to merge into any group with more than one member. lora_bank.py in LAION-AI/Humaneness-Voice-Demo-Server does this and asserts on the merge path.

---

One small LoRA per vocal-burst class — chuckle, sigh, gasp, groan, whimper, giggle, yawn, whistle, sob — plus one general adapter trained on burst-bearing clips of any class. All 71 are rank-16 PEFT adapters for laion/moss-tts-local-transformer-4.55b-voice-acting-v2-sft3.

⚠️ Read before loading

1. Do not merge. These adapters target audio_lm_heads.0…11, and in this architecture audio_lm_heads.N.weight is audio_embeddings.N.weight — one tensor, weight-tied. A merge writes the LoRA delta into the audio embedding table as well as the output head. Measured: after a merge both tensors had moved by exactly 6.103515625e-05. Load them as PEFT adapters and leave them unmerged.

2. Unevaluated. All 71 trained cleanly — not one non-finite loss or gradient in 84,525 optimiser steps — but no listening test, no automatic scoring and no A/B against the bare base has been run. What follows describes what they were trained on, which is measured; not what they do, which is not.

The base model

These adapters belong to SFT3 and to nothing else — the prompt format changed between rounds, so an adapter trained against SFT3 will not behave correctly on v2 or v1.

Usage

import torch
from transformers import AutoModel, AutoProcessor
from peft import PeftModel

BASE = "laion/moss-tts-local-transformer-4.55b-voice-acting-v2-sft3"
REPO = "laion/moss-va-sft3-vocal-burst-lora-adapters"

proc  = AutoProcessor.from_pretrained(BASE, trust_remote_code=True)
model = AutoModel.from_pretrained(BASE, trust_remote_code=True,
                                  dtype=torch.bfloat16, attn_implementation="sdpa").cuda()

model = PeftModel.from_pretrained(model, REPO, subfolder="adapters/chuckle",
                                  adapter_name="chuckle")
model.eval()
# model.merge_and_unload()   # <-- never: the audio heads are weight-tied

Dialling the effect

PEFT stores a per-adapter scaling (alpha / r = 32/16 = 2.0). Multiplying it is an exact merge weight: w = 0 reproduces the bare base, w = 1 is the adapter as trained, w > 1 extrapolates.

def set_lora_scale(model, w, adapter="chuckle"):
    for m in model.modules():
        if hasattr(m, "scaling") and adapter in getattr(m, "scaling", {}):
            if not hasattr(m, "_base_scaling"):
                m._base_scaling = dict(m.scaling)
            m.scaling[adapter] = m._base_scaling[adapter] * w

On the emotion adapters of the same family and rank a sweep over w ∈ {0, 0.25 … 2.0} moved the target score monotonically on 27 of 31 adapters, with quality holding to about w = 1.5. That sweep has not been repeated here.

Prompting

Trained with the project's v3 prompt library, PROMPT_FORMAT_HASH = 073aeb09dc923376. Prompt the adapted model exactly as you prompt the base — the adapter changes delivery, not format.

<user_inst>
- Reference(s):
{reference}
- Instruction:
{instruction}
- Tokens:
{tokens}
- Quality:
{quality}
- Sound Event:
{sound_event}
- Ambient Sound:
{ambient_sound}
- Language:
{language}
- Text:
{text}
</user_inst>

Instruction is two parts:

• GENERAL: one sentence — age and gender, delivery (arousal, tempo, tension, variability), timbre, speech quality, affect, style, recording quality, then genuineness x/6, vocal-burst blend x/10, the total duration and the language.
• SCRIPT: the timed script. Every sentence carries [D seconds duration]; every gap of ≥ 0.20 s carries [G seconds pause]; a vocal burst is (label, D seconds); and a delivery direction is a parenthesis with no number, e.g. (bitter, almost spat).

Tokens is the length in audio frames: the tokenizer runs at 12.5 fps, so tokens = round(seconds × 12.5) and one frame is 80 ms.

A burst inside speech, from the nervous_giggle bucket:

SCRIPT:
(nervous giggle, 0.1 seconds) (faintly awed — trying not to show how much this moves you, smile
barely audible, warm, composed, private, natural, not cinematic; energised, bright)
[0.2 seconds duration] Eine [1.4 seconds duration] medizinische Untersuchung hier...
[1.5 seconds pause] (awed, faintly, energised) [1.6 seconds duration] ...

A burst on its own, from the chuckle bucket — the isolated-burst half of the training data:

GENERAL: An isolated vocal burst with no speech around it, close-miked and dry;
genuineness 3.2/6; vocal-burst blend 4.4/10; 0.6s, EN.
SCRIPT:
(neutral, natural delivery, exactly as this voice normally speaks) (chuckle, 0.6 seconds)
- Tokens:
6

How the training data was built

Each class adapter mixes up to three sources, and the mix is recorded per class in manifest.json and in the table below.

1. Detector-labelled, embedded in speech. Corpus clips whose burst detector found that class inside real speech, gated on vocal-burst blend and genuineness percentile ≥ 0.80 (relaxed to 0.50, then to none, when a class was too small — the realised gate is in the table), balanced over age × gender taken from the row's own VoiceNet vector. This is the burst in context, which is where it has to work.

2. Scripted-cue, embedded in speech. The detector's vocabulary has 44 labels; the parenthesised cues written into the corpus text have 198. Rare classes — nervous giggle, trembling whimper, convulsive sob, pain moan — exist only in the second. A cue is a synthesis instruction, not an observation (39.4 % were never confirmed by the detector), so a cue alone is not evidence. These rows require a cue and a detected burst on the same clip.

Which detected burst the cue names is settled by positional alignment: when a clip's number of cues equals its number of detected bursts, the i-th cue names the i-th burst. That was checked, not assumed — restricted to cues the detector itself can emit, cue[i] == detector_label[i] on 594,830 of 629,074 positions (0.946), against 0.757 when the detector's labels are shuffled inside the same clip. Only the matching span is relabelled; the others keep the detector's label, because the clip really does contain them.

These rows come almost entirely from the synthetic voice-profile arm — real speech transcripts carry no stage directions. For trembling_whimper, 2,582 of 2,584 candidates were synthetic.

3. Isolated bursts. The standalone burst corpus, already MOSS-encoded at 12.5 fps: 97,246 clips over 77 classes. Gated on the detector confirming a burst, ≥ 2 frames, and genuineness above the class median; ranked by genuineness and balanced over provenance and duration decile.

Why the blend score is not the gate here. Blend measures how well a burst sits inside speech. A clip that is nothing but the burst has no speech to sit in, and the scorer collapses accordingly: surprised_gasp keeps 34 of 3,213 clips at blend ≥ 5/10. That is a statement about the metric, not the clips. The scales differ too — the corpus column is 0–1 (median 0.913) while the isolated-burst file carries the raw scorer output on 0–10, so a percentile gate is a no-op there. The blend gate is applied where it means something: to the embedded rows.

The classes

Totals: 70 class adapters, 67,540 training rows, 84,525 optimiser steps, ~16 GPU-hours on GH200. Plus blend_genuine_emotional, the general adapter: 29,594 rows, 36,995 steps, 7.4 GPU-hours, selected on burst quality first (blend and genuineness percentile ≥ 0.80) and emotional strength second (the row's own strongest emotion head ≥ 0.90), with low mumble and ahem capped so they cannot dominate — untamed they are 45 % and 31 % of all detections.

43 classes have detector-labelled rows, 27 draw on scripted cues, and 12 are built from isolated bursts alone.

Seven classes were not built

Below 100 usable rows after the gates, all mouth noises the detector never emits and the script never names: chewing_noises (57), click_one_s_tongue (73), kissing_noises (56), licking_sound (85), person_whistling_playfully (95), slurping_noises (80), sucking_noise (89).

Training

Two library fixes were needed for the isolated bursts and are worth knowing if you rebuild this data. anno.render could not render a wordless clip at all — the branch that treats a clip with no words as "the events are the bursts" was unreachable behind a guard, so a clip that is only a chuckle rendered as empty text instead of (chuckle, 0.6 seconds). And burst dropout (p = 0.10) now never fires on a wordless row, because dropping the only burst would teach that an empty script should produce one. Neither change touches a row that has words, and PROMPT_FORMAT_HASH is unchanged at 073aeb09dc923376.

Limitations

1. Unevaluated. Training loss is not evidence of an audible effect.
2. The labels are model outputs — the burst detector, the blend scorer and the genuineness scorer are all models, with their own errors and speaker priors.
3. Class sizes span 30×, from 111 rows (smacks_lips) to 1,600. They are not equally trained at the same epoch count.
4. The rare classes lean synthetic. Their identity comes from a stage direction in a synthesised script, validated by a detector that cannot name that class.
5. Classes overlap. sobs, quiet_sob and convulsive_sob are three adapters over overlapping material, and the adapters are not orthogonal.
6. English and German only.

Provenance

Trained on JUPITER (Jülich Supercomputing Centre, project reformo), GH200, August 2026. Bucket construction vb_buckets.py, trainer train_bucket_loras.py; both, with the full research log, in LAION-AI/Voice-Acting-Pipeline-WIP. manifest.json carries per-adapter rows, steps, wall-clock and the source composition.

<!-- recipes-and-prompting -->

Recipes, prompting and the rest of the stack

A newer set supersedes this one for the classes it covers: laion/moss-va-sft3-vocal-burst-lora-adapters-v2 holds 45 per-class adapters, 30 group adapters and the ablation arms. This set remains the fallback for anything not there.

Where the recipes live

Writing the cue

Brackets carry meaning in this format, and two rules catch nearly everyone out.

Cues are always written in English, even when the spoken line is German. This is how the training corpus is written — German rows read Das zerreißt einen einfach, weißt du? (relief sigh) — so a German cue is out of distribution.

Never write a number inside a bracket. The durations are computed for you and inserted afterwards: [N.N seconds duration] before each speech segment, [N.N seconds pause] for gaps, (label, N.N seconds) for each burst, summed at 12.5 frames per second.

<!-- /recipes-and-prompting -->