cmubye-phoneme-encoder

CMUBYE phoneme encoder

wav2vec2-XLS-R-300M phone-CTC encoder, fine-tuned on speechocean762 for pronunciation assessment. Drop-in for vitouphy/wav2vec2-xls-r-300m-timit-phoneme; same 42-token IPA vocabulary.

Results

Held-out speechocean762 test split, zero speaker overlap with train.

Phone and word level exceed commonly published results on this benchmark (~0.60 / ~0.55).

Training

Targets are what the annotators heard, not what the prompt said — substitutions use the realised phone, deletions drop out, distortions keep the canonical. Training on canonical phones instead teaches the model to emit the expected phone regardless of the audio, which makes goodness-of-pronunciation measure nothing while CTC loss still looks healthy.

10 epochs, all 24 transformer layers unfrozen, batch 8 x accum 2, lr 2e-5.

Usage

from transformers import AutoModelForCTC, AutoProcessor

proc = AutoProcessor.from_pretrained("Pransfrance/cmubye-phoneme-encoder")
model = AutoModelForCTC.from_pretrained("Pransfrance/cmubye-phoneme-encoder")

The encoder alone gives frame-level phone posteriors. Alignment, GOP, error decoding, completeness and the calibrated scoring head live in CMUBYE.

Limitations

Fine-tuned on Mandarin-L1 English learners, ~46% of them children aged 6-8. It transfers usefully to other L2 English but is not adapted to any specific target population; re-running the fine-tune on in-domain audio is the single highest-value improvement available.