moss-va-sft3-burst-synth-loras

Vocal-burst LoRA adapters trained on synthesised data

Rank-16 PEFT adapters for laion/moss-tts-local-transformer-4.55b-voice-acting-v2-sft3.

These are a replacement set for six classes of laion/moss-va-sft3-vocal-burst-lora-adapters. The recipe is that repo's recipe verbatim — rank 16, alpha 32, 5 epochs, lr 1e-4, batch 4, same prompt format (PROMPT_FORMAT_HASH = 073aeb09dc923376). Only the training data differs, and that data is manufactured: a real human vocal burst spliced into the middle of a real utterance.

⚠️ Read before loading

1. Never merge. These adapters target audio_lm_heads.0…11, and in this architecture audio_lm_heads.N.weight is audio_embeddings.N.weight — one tensor, weight-tied. A merge writes the LoRA delta into the audio embedding table as well as the output head, and the corruption is not recoverable by unloading. Measured: after merge_and_unload() both tensors had moved by exactly 6.103515625e-05. Load as PEFT adapters and set module.scaling[name] instead. The three-line check is in Never merge below.

2. The burst is in a different voice from the speech around it. This is a real property of the training data, not an artefact of one bad row, and it is the largest known weakness of this corpus. See Voice conversion was tested and rejected.

3. The labels come from a classifier, not from a human. See The circularity caveat. The source dataset's label agrees with our evaluation detector 3.4 % of the time.

What is in here

adapters/<class>/ — one subfolder per class, each holding adapter_config.json and adapter_model.safetensors. Identical layout to the repo these replace, so switching is a one-string change. manifest.json carries the per-class data and training record.

RESULTS.md holds the full before/after tables with sample sizes.

What it does, in one table

Old adapter vs new adapter, same prompts, same seeds, paired at the prompt level, inline cue, merge weight w = 1.0, n = 10 paired prompts per cell (three samples each, averaged within a prompt before the t). Strict and family-relaxed are shown together because they say different things and only one of them moves.

* = |t| above the two-sided 5 % point on n−1 d.f.

Read the wrong-burst column in the same breath as the gain, not after it. Four classes gain more at family level than they lose to wrong bursts; shriek gains +0.033 and pays +0.367, and on the solo cue frustrated_groan gains +0.200 and pays +0.333. The earlier single-class measurement (n = 9) had the wrong rate rise +0.481 against a family gain of +0.259 — larger than the gain itself. What these adapters reliably do is make the model emit a burst; making it emit the right one is achieved in four of twelve class × cue cells — those four being the cells where a significant family-level gain also exceeds the rise in wrong bursts (frustrated_groan, cackle, snicker, sniff, all inline).

The strict metric is +0.000 almost everywhere and that is not hidden. It stays there because the detector renames bursts inside their own family: for frustrated_groan the label the new adapter produces is Exhausted Groan (3 → 25 detections at w = 1.0), which is the same label the detector gives the real clean Frustrated Groans this corpus was built from. The strict metric is measuring a naming convention as much as a sound. Full per-label breakdown in RESULTS.md.

Replication. frustrated_groan was measured on two independent prompt sets. All four cells are positive; the star moves between cues (burst_dose n = 9: solo +0.259*, inline +0.111 ns; burst_dose2 n = 10: inline +0.533*, solo +0.200 ns). Two sets agreeing on the sign is worth more than either t-value; the honest effect size is a positive family-level shift of roughly +0.1 to +0.5, direction reproduced, magnitude not pinned down.

Which classes were attempted, and which were skipped

Built and shipped (6): frustrated_groan, cackle, shriek, snicker, cough, sniff.

Attempted and skipped (5): clicks_tongue, heavy_breathing, soft_whistle, spitting, tongue_click. All five were planned and all five had their source bursts classified; none was built. They are not measurable with this instrument, not failures — vf_compare.family_of returns None for every one of them, so neither the strict nor the family-relaxed metric can score them at all, and a perfectly built corpus would still read +0.000 on both. The detector maps them onto giggles, mumbles and sighs instead (soft_whistle → Low Mumble 327/446; tongue_click → Childlike Giggle 81 / Breathy Giggle 76; clicks_tongue → Ahem 81; spitting → no_burst 24 / Breathy Giggle 24; heavy_breathing → Contented Sigh 67 / Exhausted Groan 45).

How the data was made

Each training row is one assembled clip: speech A → gap → burst → gap → speech B.

1. The speech. Both halves come from the project's own SFT corpus, filtered to genuineness ≥ 0.75 and to utterances whose top-3 emotions by corpus percentile include one of the class's affinities (frustrated_groan → Impatience_and_Irritability, Anger, Bitterness, Disappointment, Contempt, Helplessness, Distress, Fatigue_Exhaustion; shriek → Fear, Distress, Astonishment_Surprise, Pain, Elation; cackle → Amusement, Malevolence_Malice, Triumph, Teasing, Contempt; snicker → Amusement, Teasing, Contempt, Malevolence_Malice; cough → Embarrassment, Fatigue_Exhaustion, Distress, Concentration; sniff → Sadness, Disappointment, Shame, Disgust, Helplessness). Percentile, not raw score — the 40 emotion heads are not comparable raw, so a raw "top emotion" would be an Interest detector. A and B are the same speaker, so the speech either side of the burst is continuous.
2. The burst comes from laion/vocal-bursts-clean.
3. The cuts. A, the burst and B are each trimmed to their own quietest 20 ms window within ±150 ms of the join, so every cut lands in a local RMS minimum rather than mid-phoneme. The pieces are joined with 10 ms equal-power crossfades and gaps drawn per row from 80–400 ms.
4. The burst is levelled independently — speech to −20 dBFS, burst to −20 dBFS plus a random −5…+2 dB — so the model cannot learn "the burst is the loud bit" as a shortcut.
5. The annotation is taken from the assembly, not from the plan: the burst's span is read back at the midpoint of each crossfade, and the row is rendered, parsed back and dropped if the label, duration (±0.5 s) or position (±1.5 s) does not survive the round trip. 97.7–98.8 % of rows survive per class.
6. Targets are encoded with the same MOSS codec the corpus uses (48 kHz, two-channel, first 12 of 32 quantisers), verified against the corpus's own stored codes.

Reuse, and how it was varied

There are far fewer clean bursts than rows — 230 groans for 2,000 rows is 8.7× — so every source burst appears several times. Reuse is capped at 10× (n_rows = min(2000, pool × 10), which is why sniff has 330 rows and not 2,000), and the repeats are made non-identical along four axes:

• speaker — a burst is never paired twice with the same speaker (330–1,605 distinct speakers per class; sniff has 330 rows and uses a different speaker for every one);
• carrier utterance — utterance reuse is 1–3, mean 1.0–1.3, so the surrounding sentence differs;
• position and gaps — the two gaps are redrawn per row from 80–400 ms, so the burst sits at a different point in a differently-timed clip;
• level — the −5…+2 dB jitter is redrawn per row.

Both languages are held at exactly 50/50. German cannot meet the strict top-3 emotion criterion (its pool is smaller than the slots it must fill, because one German source has no speaker identity at all), so German alone is widened to top-5; English is always strict. This is recorded per class in manifest.json as relaxed_langs.

Detector-aligned selection

For the five classes other than frustrated_groan, the source bursts were filtered by our own evaluation detector: a burst is kept only if the detector's top-1 label falls in the class's own family. This keeps 93 % of cackle, 99 % of snicker, 85 % of shriek, 24 % of cough and 18 % of sniff. The cue is unchanged — the script still says (cackle) — so only the audio behind the cue changes.

This buys agreement with the instrument, not ground truth. The same detector loses 74 of 264 burst-length clips to its own locator stage and disagrees with the source dataset's labels 96.6 % of the time. Aligning the corpus to its vocabulary is what lets a real effect become visible in the metric; it establishes nothing about whether the detector is right about the sound. Only listening settles that.

frustrated_groan was built before this filter existed and uses the unfiltered pool. Its numbers are therefore the more conservative ones.

Voice conversion was tested and rejected

The obvious way to remove the voice mismatch is to convert each burst into the surrounding speaker's voice. That was tried first, with Chatterbox VC, and it failed a pre-registered gate on 264 burst pairs plus 24 speech controls:

The converted clip is still a vocal burst 99.6 % of the time — it is simply a different one. The dominant transformation is toward mumbled speech (Ahem → Low Mumble, Exhausted Groan → Low Mumble, Childlike Giggle → Low Mumble), which is what a model trained to render speech in a target voice does to a non-speech input; it is also why the speech controls' error rate doubles. Survival tracks how speech-like the burst is: soft_whistle 0.71 and snicker 0.67 against heavy_breathing 0.12, spitting 0.17, tongue_click 0.21.

So the bursts here are spliced unconverted. In every training row the burst is a different speaker's voice from the speech around it. A model can in principle learn "the burst is where the voice changes"; the level jitter removes loudness as a cue, and 126–959 rows per class carry a same-speaker reference clip, but the mismatch itself is real and is not removed.

The circularity caveat

laion/vocal-bursts-clean's labels are the top-1 of laion/vocalburst-classifier-single, a sibling of the detector this project evaluates with. Training on its labels and scoring with our detector is partly self-confirming.

It is also weaker than it sounds. On 264 clean source clips the evaluation detector names the source dataset's own label 3.4 % of the time and something in the right family 34.1 % of the time. Systematic confusions, not noise: Shriek → Scream 20/24, Frustrated Groan → Exhausted Groan 14/24, Clicks Tongue → Ahem 13/24, Soft Whistle → Low Mumble 17/24.

Five further classes were not built because the detector has no vocabulary for them at all — soft_whistle (1 of 446 source clips family-consistent), heavy_breathing (9 of 200), spitting (0 of 139), clicks_tongue (0 of 221), tongue_click (0 of 330). Those are reported as not measurable with this instrument, not as failures.

Usage

import torch
from transformers import AutoModel, AutoProcessor
from peft import PeftModel

BASE = "laion/moss-tts-local-transformer-4.55b-voice-acting-v2-sft3"
REPO = "laion/moss-va-sft3-vocal-burst-lora-adapters-synth"

proc  = AutoProcessor.from_pretrained(BASE, trust_remote_code=True)
model = AutoModel.from_pretrained(BASE, trust_remote_code=True,
                                  dtype=torch.bfloat16, attn_implementation="sdpa").cuda()

model = PeftModel.from_pretrained(model, REPO, subfolder="adapters/cackle",
                                  adapter_name="cackle")
model.eval()
# model.merge_and_unload()   # <-- NEVER. See below.

Prompt the adapted model exactly as you prompt the base — the adapter changes delivery, not format. The burst cue is the class name in parentheses, e.g. (cackle), placed in the script where the burst should happen.

Dialling the effect

PEFT stores a per-adapter scaling (alpha / r = 32/16 = 2.0). Multiplying it is an exact merge weight without touching any tensor: w = 0 reproduces the bare base, w = 1 is the adapter as trained, w > 1 extrapolates.

def set_weight(model, w, adapter):
    """Exact merge weight, applied to PEFT's scaling. Never mutates a weight tensor."""
    for m in model.modules():
        if hasattr(m, "scaling") and adapter in getattr(m, "scaling", {}):
            if not hasattr(m, "_base_scaling"):
                m._base_scaling = dict(m.scaling)
            m.scaling[adapter] = m._base_scaling[adapter] * w

Recommended weight: w = 1.0. Evidence, pooled over the six classes at n = 60 paired prompts: the family-relaxed rate is flat at w = 0.25 (+0.011, t +0.41) and w = 0.5 (+0.050, t +1.84) and only moves at w = 1.0 (+0.289, t +6.04 inline; +0.128, t +3.10 solo), while the word error rate stays inside its own 3σ paired noise floor (+0.104) at every w ≤ 1.0 on the inline cue. Do not go above 1.0: a separate dose study on this adapter family measured w = 1.5 as unusable (pooled −0.174, t −4.70, ΔWER +0.329), and on frustrated_groan these adapters breach the ΔWER floor at both w = 1.25 (+0.114) and w = 1.5 (+0.226). On the solo cue even w = 1.0 breaches it (+0.141 pooled), so if you generate bursts in isolation rather than inside a sentence, back off to w = 0.5 and expect no measurable gain.

Never merge

audio_lm_heads.N.weight and audio_embeddings.N.weight are one tensor. Confirm it on your own checkout before trusting any merge in this codebase:

h = model.base_model.model.audio_lm_heads[0].weight
e = model.base_model.model.audio_embeddings[0].weight
assert h.data_ptr() == e.data_ptr(), "not tied — check your build"
print("tied:", h.data_ptr() == e.data_ptr())   # True

Because they share storage, merge_and_unload() adds the LoRA delta to the audio embedding table as a side effect. The embedding table is what turns codec tokens into vectors, so the damage shows up as degraded audio everywhere, not just on burst cues, and unloading does not undo it. Use set_weight above.

Files

adapters/<class>/adapter_config.json
adapters/<class>/adapter_model.safetensors
manifest.json     per-class data + training record
README.md
RESULTS.md        before/after tables, with n

Limitations

• The burst is in a different voice from the surrounding speech (see above). This is the big one.
• Labels are classifier-derived, not human-verified.
• Only six classes. Five more were attempted and abandoned as unmeasurable.
• On the solo cue (a burst generated on its own rather than inside a sentence) the gain is weaker and the word error rate breaches its noise floor at w = 1.0. These adapters are for bursts placed in speech.
• Two family maps exist in the build tree and they are not identical: source bursts were selected under a map with a nose family (sniff/snort/sneeze) while the metric scores sniff under the wider breath family. The selection was therefore stricter than the metric, not looser, so sniff's +0.233 is not inflated by it — but only 33 source bursts survived that filter.
• sniff is trained on 33 distinct source bursts and cough on 83. Treat both as preliminary.
• The strict metric — does the detector emit the exact class name — is +0.000 for these adapters at every weight, exactly as it is for the adapters they replace. The gain that is measurable is at family level. Both numbers are in RESULTS.md; neither is hidden.