Downloads · 30 days
1.4K
75% of all-time downloads
mrfakename/w2v-bert-2.0-music-lyrics-ctc
w2v-bert-2.0-music-lyrics-ctc is a automatic speech recognition model from mrfakename. Use it when you need speech turned into text. It is set up for transformers. The card lists the license as mit.
A character-level CTC lyrics model for full music mixes, i.e. vocals with the band, no source separation. It's fine-tuned from w2v-bert-2.0-music, with the encoder unfrozen. It is multilingual: the 921-token vocabular…
Downloads · 30 days
1.4K
75% of all-time downloads
All-time downloads
1.8K
Public
Parameters
581M
2.3 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors2.3 GB · 100%
From the Hugging Face model README
A character-level CTC lyrics model for full music mixes, i.e. vocals with the band, no source separation. It's fine-tuned from w2v-bert-2.0-music, with the encoder unfrozen. It is multilingual: the 921-token vocabulary covers Latin, Cyrillic, Greek, Arabic, Hebrew, Thai, kana/kanji and Hangul (decomposed into jamo).
Uses:
import torch
from transformers import Wav2Vec2BertForCTC, Wav2Vec2BertProcessor
processor = Wav2Vec2BertProcessor.from_pretrained("mrfakename/w2v-bert-2.0-music-lyrics-ctc")
model = Wav2Vec2BertForCTC.from_pretrained("mrfakename/w2v-bert-2.0-music-lyrics-ctc").eval()
# 16 kHz mono; up to ~30 s per call (see "Long audio" below)
inputs = processor(audio_16khz_mono, sampling_rate=16000, return_tensors="pt")
with torch.no_grad():
logits = model(**inputs).logits # (1, frames at 50 Hz, 921)
text = processor.batch_decode(logits.argmax(-1))[0]
unicodedata.normalize("NFC", text) to recompose syllables.| is the word delimiter / filler.| between
words and at both ends of the target sequence.[Chorus], (backing vocals)) are not transcribed.The model was trained on windows of up to 30 s. For longer audio, run overlapping windows (e.g. 30 s with 2 s of context on each side), keep the centre of each window, and concatenate the logits before decoding or aligning.
aux_heads.safetensors holds two linear heads that operate on the final hidden states
(output_hidden_states=True, last layer):
vad.{weight,bias}: 1 logit per frame, vocal activity.f0.{weight,bias}: 361 outputs per frame. Index 0 is the voicing logit; indices 1..360 are vocal pitch
bins of 20 cents each (bin b → 10 * 2 ** ((20 * b + 1997.38) / 1200) Hz, ≈32 Hz–2 kHz).Fine-tuned from w2v-bert-2.0-music with weak supervision from whole-song lyrics:
The vocal-activity and F0/voicing heads were trained jointly as auxiliary losses (weight 0.1 each).
Held-out songs, greedy decoding, no language model:
| CER | |
|---|---|
| Line-aligned chunks | 12.8% |
| Arbitrary windows (not cut at line boundaries) | 13.3% |
The reference lyrics are imperfect: the model often recovers words the references miss, so the true error is somewhat lower.
Forced-alignment timing: a word/syllable onset evaluation will be added here.