xlsr-twi-codeswitch-ipa

xlsr-twi-codeswitch-ipa

A phoneme recogniser for Ghanaian English–Twi code-switched speech.

KoelLabs/xlsr-english-01 fine-tuned on ghananlpcommunity/Ghana_English-Twi_Code-switching_Speech-ipa, which is KasaSpeech with an ipa column produced by ghanag2p-uni.

The model outputs phonemes in the same 79-token inventory as its base model. Nothing was added to the vocabulary and the CTC head was not resized: the training targets were mapped into that inventory first, so every pretrained output row kept its meaning.

Results

Phoneme error rate (PER), lower is better. The test split was held out completely during training — it was read once, by this evaluation.

Test errors break down as 17,783 substitutions, 22,139 deletions, 3,678 insertions over 155,838 reference phonemes.

PER by utterance length on test:

Usage

import json, torch, soundfile as sf
from huggingface_hub import hf_hub_download
from transformers import Wav2Vec2ForCTC, Wav2Vec2FeatureExtractor

model = Wav2Vec2ForCTC.from_pretrained("ghananlpcommunity/xlsr-twi-codeswitch-ipa").eval()
fe = Wav2Vec2FeatureExtractor.from_pretrained("ghananlpcommunity/xlsr-twi-codeswitch-ipa")
vocab = json.load(open(hf_hub_download("ghananlpcommunity/xlsr-twi-codeswitch-ipa", "vocab.json")))
inv = {i: u for u, i in vocab.items()}

speech, sr = sf.read("utterance.wav")      # must be 16 kHz mono
x = fe(speech, sampling_rate=16000, return_tensors="pt")
with torch.no_grad():
    ids = model(x.input_values).logits.argmax(-1)[0].tolist()

# CTC decode: collapse runs, drop the blank (id 0)
out, prev = [], None
for i in ids:
    if i != prev and i != 0:
        out.append(inv[i])
    prev = i
print(" ".join(out))       # e.g. "ɛ j ɛ d e n o b u t ɪ ʔ l l t ɾ j"

Training

Clips over 20 s were excluded from training only; the validation and test splits were evaluated in full, long utterances included.

Caveats

• Targets come from a grapheme-to-phoneme rule set, not human phonetic annotation. The model learns to predict what ghanag2p-uni would produce for the reference transcript, so it inherits that system's conventions — and its errors.
• Everything is phonemised under Twi rules, English stretches included, so English words are rendered as a Twi speaker's phonology rather than with an English G2P.
• The target inventory is lossy against real Twi: tʰ collapses to t, vowel length is dropped, ç becomes ʃ and ɲ becomes n j.

Credit

Speech and transcripts are from KasaSpeech (Kenneth Dotse). Licence AGPLv3, as the base model and source corpus.