echora-mms-300m-multilingual-lyrics-forced-aligner

Echora MMS 300M Multilingual Lyrics Forced Aligner

A multilingual forced-alignment model for producing timed lyrics and karaoke segments from music audio and supplied lyric text.

Model Details

• Model type: Wav2Vec2 / MMS CTC forced aligner
• Base model: facebook/mms-300m
• Languages: English, Japanese, Korean, Spanish, Indonesian, Urdu, Hindi, and mixed-script lyrics
• Parameters: Approximately 300M

Versions

• v8 (this revision): fine-tuned from v6 on separated vocal stems, with background vocals that overlap the lead restored to the training data.
  • Trained with a second output head for background vocals sung over the lead. The weights include it (stream2_head in the config); Wav2Vec2ForCTC loads the lead head and reports the extra weights as unused, which is expected.
  • Works best on a separated vocal stem; Echora separates vocals before aligning.
  • Korean and Japanese input expectations are unchanged from v6.
• v6: revision 1f241323a4f5d961160ae7c7c88e5c7d27f32f59, fine-tuned from v5 with a revised dataset and training objective.
  • Background vocals sung alone are aligned; background vocals overlapping the lead are no longer forced onto lead timing.
  • Korean lyrics are expected romanized by pronunciation (liaison, nasalization and other sound changes applied), not letter by letter. Echora's pipeline does this automatically.
  • Japanese lyrics use kana readings resolved from dictionary data where possible.
• v5: earlier revision (b46485a5d814dc26e3511cece3ccc98ebba2e9d0). It expects letter-by-letter Korean romanization; pin it if your pipeline produces that.

Intended Use

Use this model to align user-supplied lyric text with audio and produce approximate word or syllable timestamps for personal karaoke and lyric-display projects.

Use Restrictions

Personal, non-commercial use only.

Do not use this model for commercial products, paid services, redistribution, or training other models. You are responsible for ensuring that you have the necessary rights to the audio and lyric text you process.

No permission is granted for copyright-sensitive, rights-restricted, or otherwise unauthorized use.

Limitations

• Timing quality varies depending on vocal clarity, instrumental density, recording quality, language, and lyric accuracy.
• Repeated phrases, long vocal holds, ad-libs, rap, vocal effects, and overlapping vocals may produce inaccurate boundaries.
• Runs of identical interjections ("ooh, ooh", "oh-oh-oh") can be placed late, with the first held over the rest.
• Romanized lyrics may have weaker syllable boundaries than lyrics written in their native scripts.

Getting Started

from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor

model_id = "hcX02/echora-mms-300m-multilingual-lyrics-forced-aligner"

processor = Wav2Vec2Processor.from_pretrained(model_id)
model = Wav2Vec2ForCTC.from_pretrained(model_id)

This repository provides model weights only. Use a compatible forced-alignment pipeline to combine acoustic emissions with the supplied lyric text.

Training

This model was fine-tuned for multilingual lyric forced alignment using timed lyric supervision, with validation and test songs held out by artist and matched to the exact recording.

Training configuration, checkpoint selection, and evaluation details are maintained by Echora.

Evaluation

Validation uses lyric-frame accuracy and alignment loss against held-out timed lyric segments. These metrics measure agreement with timed labels and do not guarantee perceptually correct karaoke timing.

Contact

Open an issue in the Echora repository.