vocal-burst-lora-adapters

Vocal-Burst LoRA Adapters (Gemini-Confirmed, 64 classes)

64 PEFT/LoRA adapters for laion/moss-tts-local-transformer-4.55b-voice-acting-v2, one per non-verbal vocal-burst class (laughs, sighs, gasps, whistles, breathing, grunts, moans, etc.), trained on classifier-filtered DramaBox audio and selected from a pool of 77 trained classes using a listening evaluation, not the automatic metric alone.

Each class ships as a single adapter directory (its own adapter_model.safetensors + adapter_config.json) at the rank and training epoch that scored best. classes.json has the full per-class metadata; inference.py is a runnable example of how to load the base model and one adapter and generate a clip.

👉 Getting the burst to actually happen inside a sentence

The adapters are usually wanted for a burst mid-utterance, not in isolation. That case was measured separately over 1,440 generations (8 emotion families × 60 classes, same carriers, same seeds, same inline burst request):

🎧 Listen to all four cells side by side

• Keep the inline tag in the SCRIPT. The prompt alone already lands the burst about a quarter of the time; the adapter multiplies that base rate rather than replacing it.
• An emotion adapter on its own suppresses the burst (23.6 % → 16.7 %, below the no-adapter baseline). If you want a burst inside emotional speech, the burst adapter has to carry it.
• Dose: λ = 0.75 is the knee (64.8 % presence, best blend, peak genuineness); λ = 1.0 buys a few more points of presence for a little more blend. λ = 0.25 is not worth generating — at 27.3 % it is within noise of the prompt-only baseline.
• Stacking: keep any emotion adapter at ≤ half the burst dose. [email protected] + [email protected] was the best of a 2,304-generation sweep; a half-merged pair was the worst cell measured.
• Sexual_Lust and Pain classes peak at λ = 0.75 and get worse at 1.0. Six of eight families are monotone in dose; those two are not.
• Five classes never fire mid-sentence at any dose: Hiss, Kissing Noises, Lip Smack, Person Whistling Playfully, Slurping Noises. Generate those as isolated events.
• Need it in every take? At 72 % per take, 4 candidates give you ~99 %. Select on burst presence, not on the burst classifier's label — the classifier cannot reliably name a burst embedded in speech (locator timing is reliable, the label is a weak prior).

Full dose curves, the stacking interaction and the evaluation method: manual chapter.

Where everything lives

One result worth knowing before you pick a merge scale

Merging an adapter harder does not monotonically improve things. Measured on cross-lingual voice cloning, ECAPA speaker similarity to the reference falls as the emotion-adapter merge dose rises — 0.62 at dose 0, 0.57 at 0.5, 0.50 at 1.0, and −0.03 at 1.5, against an unrelated-speaker floor of 0.105. At dose 1.5 the output voice has no relationship to the reference at all. If reference-voice fidelity matters, keep emotion merges ≤ 0.5. (Burst adapters on burst-only clips behaved differently — full dose won every class there — so treat the two cases separately.)

Why 64 of 77, and why a listening judge

The training run produced 77 class adapters, each at 3 LoRA ranks (r16/r32/r64) and up to 6 epochs — about 186 GB in total. Before publishing anything, every configuration was scored two different ways:

1. An automatic evaluator (a burst locator + a vocal-burst classifier v2, blended into a composite score) run over held-out prompts.
2. A gemini-3-flash listening judge, which listened to the top-3 generated clips per class per rank and rated match-to-description on a 0/1/2 scale — with real recordings of that class and the no-LoRA base model included as blind controls, so a class scoring 2.0 could be checked against something.

The controls are the reason the judge scores can be trusted:

The judge clearly discriminates — it does not rate everything a 2 — and LoRA clips land much closer to real recordings than to the unadapted base model.

The automatic evaluator cannot recognise 26 of the 77 classes even in real human recordings — their anchor composite (the automatic score computed on real, held-out recordings of that class) is below 0.10, and six are exactly 0.000. These are mostly quiet mouth/breath sounds (whistles, hiccups, breathing, lip/kissing sounds) that the locator or classifier front-end simply doesn't register as bursts. For those 26 classes, the automatic "gain" from training measures the metric's blindness, not the adapter's quality — a class that scores worse or unchanged on the automatic metric could still be a well-trained, audibly correct adapter, and there is no way to tell from the automatic number alone.

19 of those 26 metric-blind classes were confirmed by the listening judge and are included below — including every whistle class (Wolf Whistle, Sharp Whistle, Soft Whistle, Person Whistling Playfully, Person Whistling to Get Attention), all of which the automatic metric scored as regressions relative to the base model. Listening to them tells a different story, which is the entire reason this repo uses a listening judge instead of shipping (or rejecting) classes on the automatic score alone.

Selection rule

Publish only classes with a mean LoRA judge score lora >= 1.5 (out of 2.0, averaged across all rank/epoch clips judged for that class). This produced exactly 64 of 77 classes (verified directly against gem/vb_judge.json, not assumed) — the same 64 shipped here.

For each of the 64, we ship one rank and one epoch:

• Rank: whichever of r16/r32/r64 got the higher listening-judge score for that class (by_rank in the judge data); ties are broken by the automatic composite score.
• Epoch: whichever epoch the automatic evaluator identified as best for that chosen rank (max composite over the epochs trained for that rank/class).

This means the repo ships neither "all ranks for all classes" (which would triple size for no benefit) nor a single global rank/epoch — each class gets its own best configuration.

The 13 rejected classes

These scored below 1.5 on the listening judge and are not included in this repo. They are dominated by tongue clicks, lip smacks, licking and sucking sounds — extremely short, low-energy mouth noises the model struggled to reproduce distinctly. Purr is the most severe case: it failed with only 13 usable training items, and its real recordings also score 0 on the automatic anchor metric — i.e. the source data for that class appears to be bad or mislabeled, not merely the adapter.

The 64 published classes

rank/epoch are the ones actually shipped in this repo. judge score is the Gemini listening-judge score (0–2) for that specific rank; judge mean (all ranks) is the class's overall mean used for the >=1.5 cutoff. automatic composite is the automatic evaluator's score for the shipped rank/epoch; anchor composite is the automatic evaluator's score on real recordings of that class (a low anchor means the automatic metric can't reliably "see" this class at all — see metric_blind). n train items is the number of classifier-agreed training examples actually used for that class (see caveats below).

(sorted by judge score, descending; ties broken arbitrarily)

Caveats — read before relying on these adapters

• Training data is small and heavily filtered. Across all 77 classes: 97,246 raw DramaBox items → 16,689 usable (17.2%). An item was kept only if an independent classifier agreed with its label. 15 classes have under 100 usable training items (several published classes are in this group — see the n train items column above, e.g. Swallows at 75, Lip Smack at 81, Slow Breathing at 84, Deep Breathing at 86, Kissing Noises at 94). Adapters for these classes are trained on very little data and should be treated as lower-confidence than the headline numbers suggest.
• The training-data filter is circular. Items were kept when a classifier from the same model family used elsewhere in this pipeline agreed with the human/source label. This self-reinforcing filter could systematically exclude atypical-but-correct examples of a class and/or retain examples that merely look classifiable rather than being representative, which is part of why a separate, independent (listening) judge was used for the final publish/reject decision rather than trusting the automatic pipeline end-to-end.
• The Gemini judge is a single model, not a human panel. It rated 0/1/2 on at most 9 clips per class (top-3 clips x 3 ranks), which is a strong, control-validated signal but not a substitute for human evaluation at scale. Treat class-level scores as indicative, not as ground truth.
• Not all adapters are converged. Training ran for 3 epochs by default (a handful of classes were extended to 6). Roughly 40% of classes were still improving on the automatic metric when training stopped — for those classes, the shipped checkpoint is the best one produced so far, not necessarily the best one that further training could produce.
• The automatic evaluator is not a reliable arbiter on its own for roughly a third of these classes (see metric_blind column) — those numbers are reported here for completeness and reproducibility, but the actual publish/reject and rank/epoch choices for those classes leaned on the listening judge, since the automatic composite is close to meaningless when the anchor (real-recording) score is itself near zero.

Inference

See inference.py for a complete, runnable example. Key points it documents in detail (all learned during training/eval of these adapters):

• Load the base model with attn_implementation="sdpa" — flash-attn 2.x is incompatible.
• audio_lm_heads.* and text_lm_head.weight reported MISSING on load is benign (the heads are weight-tied internally).
• Build prompts with the processor's build_user_message(text=..., instruction=..., tokens=..., language=...), not chat_template.jinja. instruction is the full "GENERAL: ...\nSCRIPT:\n..." caption; text is only the spoken words (None for vocal bursts, which the processor renders as the literal string "None" — this is expected and matches training).
• The 0.16s trap: AssistantMessage.audio_codes_list already holds decoded waveforms on the output side (Processor.decode() calls decode_audio_codes() internally). Calling decode_audio_codes() on it again silently returns 7,680 samples = 0.16s for every clip, regardless of true length. Use the tensor directly and mean over the channel axis.

from transformers import AutoProcessor, AutoModel
from peft import PeftModel
import torch

proc = AutoProcessor.from_pretrained(
    "laion/moss-tts-local-transformer-4.55b-voice-acting-v2",
    trust_remote_code=True, codec_path="OpenMOSS-Team/MOSS-Audio-Tokenizer-v2")
proc.audio_tokenizer = proc.audio_tokenizer.to("cuda").eval()
model = AutoModel.from_pretrained(
    "laion/moss-tts-local-transformer-4.55b-voice-acting-v2",
    trust_remote_code=True, dtype=torch.bfloat16, attn_implementation="sdpa").cuda().eval()

adapter = PeftModel.from_pretrained(model, "./wolf_whistle", adapter_name="wolf_whistle")

msg = proc.build_user_message(
    text=None,
    instruction="GENERAL: A person's voice. Clean studio recording.\nSCRIPT:\n(Wolf Whistle, 0.8 seconds long)",
    tokens=10, language="English")
batch = proc([[msg]], mode="generation")
out = adapter.generate(input_ids=batch["input_ids"].cuda(),
                        attention_mask=batch["attention_mask"].cuda(),
                        max_new_frames=90, do_sample=True,
                        text_temperature=0.7, audio_temperature=1.0,
                        audio_top_k=30, audio_top_p=0.95, audio_repetition_penalty=1.1)
wav = proc.decode(out)[0].audio_codes_list[0].float().cpu().numpy().mean(0)  # already decoded -- do not re-decode

See inference.py for the full, runnable version with error handling and prompt-building helpers.

Repository layout

<class_slug>/adapter_model.safetensors
<class_slug>/adapter_config.json     # one directory per class, 64 total
classes.json                          # full per-class metadata (this README's tables, machine-readable)
inference.py                          # runnable load + generate example
README.md                             # this file

License

CC-BY-4.0.

Base model / acknowledgements

Base model: laion/moss-tts-local-transformer-4.55b-voice-acting-v2. Adapters trained with PEFT/LoRA on classifier-filtered DramaBox audio. Automatic evaluation used an internal burst locator + laion/voiceclap-commercial classifier/blend/genuineness heads. Listening evaluation used gemini-3-flash as an audio judge with real-recording and base-model controls.