midashenglm-gen-wer-lora

ModelsLab/midashenglm-gen-wer-lora

A LoRA adapter for mispeech/midashenglm-gen, trained across all five of the model's capabilities — speech, sound effects, music, ambience and mixed scenes — rather than trading one for another.

Hear it

Same prompt, same seed in both arms — generate() seeds the global RNG, so the solver draws identical noise and every audible difference is the adapter.

Speech, 4-word line — "The river remembers everything." The bucket that went 13.9% → 0.0% WER; the base arm garbles it.

Speech, 16-word line — both arms intelligible; the adapter is cleaner and tighter.

SFX — footsteps on gravel. Texture and caption match improve; the adapter clip runs shorter, the one real remaining cost.

Music — slow solo piano. The adapter plays longer and closer to the caption at this seed.

Ambience — steady rain on a metal roof. Near-identical by design: this arm held.

Why

Two weak spots in the base model, measured rather than assumed.

Short lines. A four-word line scores around 42% WER where a sixteen-word one scores near zero. The failures are onsets: the model needs a moment to settle and a short line does not give it one. Prompt-side remedies make it worse, so it is a property of the weights.

Sound effects. The model's worst benchmark — AudioCaps FAD 5.01 against TangoFlux's 2.26 — which the paper attributes to training on mixed scenes rather than dedicated sound-effect corpora.

Results

Same prompt, same seed, base weights against these adapters. generate() seeds the global RNG, so the flow-matching solver draws identical noise in both arms and every difference is the adapters.

CLAP text is "does it match the caption". CLAP real is cosine to the centroid of genuine MECAT audio of that category — "does it sound like the real thing", which text similarity alone cannot see. WER cannot see either: a clip with no words in it scores 0% however badly the room tone came out.

WER improved by 5.4 points, and that part is solid. The scene arms are not.

Caption adherence or realism fell on: sfx clap_text -0.023, ambience clap_text -0.037.

Use this for speech. For beds, effects and mixed scenes, A/B it against the base model on your own prompts first — the held-out flow loss improved on every capability, and on the scene arms that did not translate.

Training

Base-model hyperparameters follow arXiv:2608.11804 §3.3 where they apply (lambda_stop = 0.01, grad clip 1.0).

Avoiding catastrophic forgetting

Four things, because one is not enough:

1. LoRA rather than a full fine-tune. Rank 32 bounds how far the weights can move at all.
2. A capability-weighted mix. The sampler draws by capability, not by pool. At natural frequency the speech buckets are ~80% of the clips and the model would simply get better at reading aloud.
3. An anchor loss. The tuned vector field is pulled towards the frozen base's on identical noise and timesteps — a trust region around the base model, stated directly rather than hoped for.
4. A per-capability gate. Held-out flow loss is tracked for each capability separately, and this checkpoint was only saved because the mean improved and no single capability regressed past 2%.

Use

from peft import PeftModel
from transformers import AutoModel
import soundfile as sf

model = AutoModel.from_pretrained("mispeech/midashenglm-gen", trust_remote_code=True)
model = PeftModel.from_pretrained(model, "ModelsLab/midashenglm-gen-wer-lora").merge_and_unload().cuda().eval()

result = model.generate(
    "<|caption|> A close, clean recording of a single narrator, with nothing behind it. "
    "<|asr|> The river remembers everything. "
    "<|speech|> A woman in her forties, warm and even, speaking clearly. "
    "<|sfx|> <|unknown|> <|music|> <|unknown|> <|env|> <|unknown|>",
    seed=11,
)
sf.write("out.wav", result["audio"], result["sample_rate"])

Tag order matters more than anything else in the prompt: <|asr|> comes before <|speech|>. Reversed, the model produces fluent unrelated speech — 14.2% mean WER against 373%.

Licences

Apache 2.0, following the base model. Training data: LibriTTS-R (CC-BY-4.0) and MECAT-Caption (CC-BY-3.0), both attribution-only.

Training code: audio-scenegen/training.