SingAlign

SingAlign — released checkpoints

Weights for SingAlign, a unified forced-alignment toolkit for singing voice. The code lives in the GitHub repository; this repository holds only the checkpoints it downloads.

Files here are named exactly as configs/models/<name>.yaml declares them, so the toolkit's resolver (modules/common/api/checkpoints.py) fetches them by name and verifies the pinned SHA256 before it puts anything on disk.

hfa-mandarin-fullweak-v1

A HubertFA alignment head over a frozen chinese-hubert-base front-end, trained in-repo with train_hfa.py. It is SingAlign's default model.

• Recipe: configs/hfa/train_fullweak_v1.yaml, data configs/hfa/binarize_fullweak.yaml.
• Training data: 51.4 h full-label (GTSinger, M4Singer, Opencpop) + 7627 h weak-label Mandarin singing (lyrics only, no phoneme durations).
• Step 15000, selected by CBER on a held-out validation set — not the final step-60000 weights, whose validation curve had already degraded.
• Phoneme inventory: 117 phones, sofa-expanded-v1 profile (identity against the toolkit's canonical Mandarin inventory).
• G2P: G2pW — no pronunciation dictionary to configure.

Results

Scored on cloudtest-verified-v1: 8 recently released original songs (90 clips, 1971 phones, 535.6 s) with hand-verified word + ph tiers, none of whose song IDs or "artist – title" pairs appear in any training manifest. Every system was fed the same gold phoneme sequence with G2P and breath detection off, so this measures acoustic alignment only.

VER20/VER50 are vlabeler edit ratios at 20 ms / 50 ms boundary tolerance and include SP; mIoU and CBER exclude it. Point estimates, no confidence intervals.

Read this as an error profile, not a ranking. This model leads on VER50, mIoU and CBER but trails upstream v0.0.7 by ~9% on VER20: it makes fewer large errors and more small ones. If your criterion is a 20 ms tolerance, the upstream model is the better pick.

Runtime dependency

The torch path does not carry its own SSL front-end. The checkpoint records hubert_config.model_path, and those encoder weights (TencentGameMate/chinese-hubert-base, ~380 MB) must be present. python scripts/download_assets.py in the toolkit fetches this checkpoint, the encoder and the G2pW model together.

Usage

git clone https://github.com/pymaster17/SingAlign && cd SingAlign
uv sync --extra pitch && source .venv/bin/activate
python scripts/download_assets.py           # this checkpoint + encoder + G2pW
python infer_one.py -a audio.wav -t "中文歌词" --out_formats textgrid,json

from modules.api import HubertFATorchAligner

aligner = HubertFATorchAligner()             # resolves to this checkpoint
results = aligner.align([{"id": "x", "audio_path": "a.wav", "text": "中文歌词"}])

hfa-english-fullweak-v2

The English model: the same alignment head over a frozen facebook/hubert-base-ls960 front-end, trained with the recipe of hfa-english-fullweak-v1 below on data whose only change is where the clips start.

• Recipe: configs/hfa/train_en_fullweak_v6.yaml (identical to v1's except the run name), data configs/hfa/binarize_en_fullweak_v6.yaml.
• Training data: 21.4 h full-label across 13 singers (as v1; Project-AIdol and NUS-48E re-cut so that a silence goes whole to one neighbouring segment instead of being split) + 1845 h weak-label English singing selected by singmos >= 3.5 only, so the corpus's silence-cut sub segments (37 %) are in the pool alongside the lyric-timed short ones.
• Step 56000, selected by CBER on held-out singers.
• Phoneme inventory / G2P: as v1 (english-arpabet-v1, G2pEn); the vocabulary file is byte-identical, so the two are drop-in for each other.

Why v2 exists

v1 puts a spurious sub-millisecond SP at the head of about a third of clips that start on the voice. The cause was traced to its weak-label pool: a wer filter silently restricted it to lyric-timed segments, three quarters of which start mid-voice, and CTC training on those teaches the shared phone channels to read onset frames as "not yet a phone". Mixing in silence-cut segments removes it: 32 % → 5.3 % of voice-initial GTSinger-en clips (bench/tools/probe_leading_sp.py; docs/experiments/leading_sp_probe.md).

Results

Same four held-out NUS-48E singers as v1 (201 clips, 0.53 h):

A tie with v1 on alignment quality; the caveat below about the weak label buying nothing measurable on clean studio audio still applies.

Batched inference is bit-reproducible

Unlike v1, this checkpoint passes the toolkit's batch-invariance gate: bench/tools/validate_hfa_batch.py on the same 48 items, batch 8, measures a maximum boundary shift of 0.002 ms against a 1.0 ms tolerance.

Runtime dependency and usage

As v1: facebook/hubert-base-ls960 must be present, and the model is named explicitly:

from modules.api import HubertFATorchAligner

aligner = HubertFATorchAligner(ckpt="hfa-english-fullweak-v2", g2p="G2pEn")
results = aligner.align([{"id": "x", "audio_path": "a.wav", "text": "english lyrics"}])

hfa-english-fullweak-v1 (superseded)

Kept because the CrawlSinger-en corpus's align_conf column was written by it; use hfa-english-fullweak-v2 for new work. The same alignment head over a frozen facebook/hubert-base-ls960 front-end — same HuBERT-base recipe as the Mandarin model, so only the encoder path and the phoneme inventory differ.

• Recipe: configs/hfa/train_en_fullweak.yaml, data configs/hfa/binarize_en_fullweak.yaml.
• Training data: 21.4 h full-label across 13 singers (GTSinger-en, ACV-001, Project-AIdol, 8 of NUS-48E's 12 singers) + 1845 h weak-label English singing (lyrics only, no phoneme durations).
• Step 54000, selected by CBER on held-out singers — not the final step-60000 weights.
• Phoneme inventory: the 39 CMUdict ARPAbet phones (stress stripped) plus SP, english-arpabet-v1 profile.
• G2P: G2pEn (CMUdict + a seq2seq for OOV) — no dictionary to configure.

Results

Scored on four held-out NUS-48E singers (201 clips, 0.53 h) — two from each mode of the corpus's F0 distribution, none of whom appears in training. The same suite selected the checkpoint, so read these as in-suite numbers.

The weak label bought nothing measurable here. The two models tie, and the gap is smaller than this run's own validation-point scatter. It is released as the default English model on the grounds that the Mandarin line measured weak label buying robustness on rough recordings — but English has no rough-domain held-out set to check that on, because the validation set is studio singing while the weak label is crawled audio. If your material is clean, expect nothing from the extra 1845 h. The full account is in docs/experiments/hfa_en_full_weak_v1.md in the toolkit.

Batched inference is not bit-reproducible

This checkpoint does not pass the toolkit's batch-invariance gate: bench/tools/validate_hfa_batch.py measures a 10.5 ms maximum boundary shift between batch-1 and batch-8 against a 1.0 ms tolerance. Scale: one boundary of one clip in 48, between content phones (SH | IY) rather than on an SP edge, so an adapter does not drop it.

Batching changes the floating-point reduction order in the linear layers, and weak-label training leaves the logits flat enough at an acoustically ambiguous boundary that the last-bit difference moves the Viterbi path. A deterministic tie-break cannot fix it — the inputs genuinely differ. Set max_batch_size=1 if you need reproducibility. For comparison, the full-label-only model measures 0.003 ms on the same items.

Runtime dependency

facebook/hubert-base-ls960, not the Mandarin encoder — the two are not interchangeable.

Usage

from modules.api import HubertFATorchAligner

aligner = HubertFATorchAligner(ckpt="hfa-english-fullweak-v1", g2p="G2pEn")
results = aligner.align([{"id": "x", "audio_path": "a.wav", "text": "english lyrics"}])

nll-hubert-ls960-v1

A breath (non-lexical) head, not an aligner: a 2.45 M-parameter CVNT that labels each 10 ms frame None / AP from the HuBERT units the aligner has already computed, and whose breaths are spliced into the finished alignment as AP without touching any other boundary. It rides on hfa-english-fullweak-v2 (or v1) as ap_detector="nll" and is selected automatically for any HubertFA aligner whose front-end is hubert-base-ls960.

• Why it exists: a breath head is bound to the SSL front-end it was trained on. The head shipped inside upstream HubertFA's ONNX bundle was trained on chinese-hubert-base units; run over hubert-base-ls960 units (same 768 width, unrelated latent space) it recalls 0.001 of the breaths in held-out English singing -- nothing. The toolkit now refuses that pairing.
• Recipe: configs/nll/train_v2.yaml, data configs/nll/binarize_v2.yaml; class-weighted CE + focal + dice, 15000 steps, step 7500 selected on frame-F1 over the annotation-consistent held-out groups.
• Training data (59.3 h): GTSinger singers whose breath annotation agrees with the upstream head's convention (ZH ×2, EN-Alto-2, EN-Tenor-1, IT ×3, ES-Soprano-1; 32.7 h) + M4Singer (26.7 h). Five GTSinger languages label the whole inter-phrase gap as breath; a head trained on all of GTSinger learns "gap ⇒ breath" and its precision collapses on any other corpus.
• Binary: None / AP. No tail-breath (EP) class.

Results

Frame-F1 at threshold 0.5, scored only inside the gold non-content regions (where a breath could be), on songs held out by title:

The Opencpop gap is a known limitation: this head still over-fires on Opencpop's silent gaps (precision 0.69 at recall 0.99). Full account, including the per-singer annotation probe: docs/experiments/nll_hubert_ls960_v1.md in the toolkit.

Usage

from modules.api import HubertFATorchAligner

aligner = HubertFATorchAligner(ckpt="hfa-english-fullweak-v2", g2p="G2pEn",
                               ap_detector="nll")   # resolves to this head

Pairing it with a chinese-hubert-base aligner raises; the Mandarin line keeps the upstream head.

Training data, attribution and terms

All checkpoints are derived from corpora with their own terms. Attribution is required by several of them; the ShareAlike terms are why this repository is CC-BY-NC-SA-4.0 rather than CC-BY-NC-4.0.

No audio from any of these corpora is redistributed here — only trained weights.

Caveats

• confidence does not transfer across models. SOFA scores land in ~[0.79, 1] and HubertFA in ~[0.67, 0.95], and the score is a top-2 pairwise margin. Any threshold inherited from another model has to be re-calibrated.
• One language per checkpoint. Each model's inventory and frozen encoder are language-specific; there is no multilingual checkpoint. The breath head is language-independent but front-end-specific.
• The weak-label half of both models' training data is scraped singing whose underlying recordings are third-party copyrighted works. The weights are released for non-commercial research for that reason, under CC-BY-NC-SA-4.0 — ShareAlike because GTSinger, in both models' full-label data, carries it. The toolkit's source code is MIT and licensed separately.

Citation

Upstream work this builds on: SOFA and HubertFA.