moss-voice-identity-loras

MOSS voice-acting v2 -- per-voice identity-recovery LoRAs

<!-- moss-tied-heads-warning -->

⚠️ Do not merge this adapter into the base weights

merge_and_unload(), merge_adapter(), and any offline "bake the LoRA into the checkpoint" script will destroy the model irrecoverably. This is not a performance caveat. Read this before you write a deployment script.

Why

This adapter targets audio_lm_heads.0 … audio_lm_heads.11 and text_lm_head — 12 of its 23 target modules. In this architecture those output heads are weight-tied to the input embeddings: tie_weights() sets

audio_lm_heads[i].weight  IS  audio_embeddings[i].weight     # the same tensor, not a copy
text_lm_head.weight       IS  transformer.embed_tokens.weight

They are one allocation with two names. So when a merge adds B @ A * (alpha/r) into the head weight, it writes that delta straight into the embedding table at the same time. The model then reads its own inputs through a matrix that has been shifted by an output-side correction. Generation does not fail loudly — it degrades into noise or into a fixed babble, and the damage is inside the checkpoint you just saved. There is nothing to unmerge afterwards, because the original values are gone.

Verify it yourself in three lines

Do not take our word for it:

m = base.model if hasattr(base, "model") else base
print(m.audio_lm_heads[0].weight.data_ptr() == m.audio_embeddings[0].weight.data_ptr())
# True  -> same storage, merging corrupts the embeddings

What to do instead

Load with PEFT and leave the adapter unmerged. Set its strength through the scaling factor:

from peft import PeftModel

model = PeftModel.from_pretrained(base, "<this repo>", adapter_name="a").to(dev).eval()
# do NOT call model.merge_and_unload()

def set_weight(model, name, w):
    """Scale one named adapter's contribution.  alpha/r is its own base scaling."""
    for module in model.modules():
        scaling = getattr(module, "scaling", None)
        if isinstance(scaling, dict) and name in scaling:
            if not hasattr(module, "_base_scaling"):
                module._base_scaling = {}
            module._base_scaling.setdefault(name, scaling[name])
            scaling[name] = module._base_scaling[name] * float(w)

set_weight(model, "a", 1.0)
model.base_model.set_adapter(["a"])      # several adapters can be active at once

This sounds identical to a merge. An unmerged LoRA computes Wx + (B @ A)x * (alpha/r), which is exactly what the merged weight W + B @ A * (alpha/r) would compute — the same arithmetic, in a different order. You give up a small amount of inference speed and you keep the ability to change the weight, stack several adapters, or turn one off. Nothing about the sound changes.

If you are stacking adapters

Set each one's scaling separately and activate them together with model.base_model.set_adapter([...]). Note that stacking is not free: in our own measurements a deep stack held audio quality but destroyed intelligibility (word error 0.063 → 0.554). Add adapters deliberately and measure.

If you maintain code that merges

A regex over module names is not enough — the reliable test is identity of storage. Group the modules by weight.data_ptr() and refuse to merge into any group with more than one member. lora_bank.py in LAION-AI/Humaneness-Voice-Demo-Server does this and asserts on the merge path.

---

One rank-4 LoRA per voice profile, trained on that voice's own takes that already sound like the voice (ECAPA speaker similarity to the voice's reference recording >= 0.60, held-out groups removed), and used to regenerate the takes of that voice that fell below the 0.40 similarity floor.

• rank 4, alpha 8, dropout 0.05, the same 23 target modules as the shipped per-voice adapters
• one stage, checkpointed and evaluated on held-out groups every epoch; the epoch shipped is the held-out-loss minimum, which varies by voice -- a fixed epoch count overfits some voices and underfits others
• merged for regeneration at scale 1.0

<voice>/report.json carries that voice's before/after numbers, the epoch curve that chose the checkpoint, and the cost columns.

Two things that must travel with any result from these adapters

1. The 0.40 floor is this project's ECAPA scale, not an established perceptual fact. An independent WavLM-tbr embedder (Orange/Speaker-wavLM-tbr, published same-speaker threshold 0.472) puts 75.4 % of the takes selected as "failures" above its own threshold.
2. Identity is bought, not free. In the reference study the identity gain cost -11 % target emotion strength and -14 % blend. Every voice's report.json reports its own cost columns.

And one about how to read a before/after

Production regenerates a single arm, so there is no per-voice control. The reference study (emolia_c1699) measured that simply re-rolling these takes with no voice adapter lifts them from 0.2 % to 14.0 % above the floor -- they are the unlucky tail of their groups, so regression to the mean alone moves them. The adapter took the same population to 43.7 %. A per-voice vs_orig delta in these reports therefore contains that re-roll component and is not the adapter's effect on its own.