wav2vec2-xlsr-53-ft-cy-en

wav2vec2 XLSR-53 — Welsh and English speech recognition — 26.10

A facebook/wav2vec2-large-xlsr-53 acoustic model fine-tuned with a character-level CTC head for Welsh speech recognition, with UK- and Irish-accented English as a secondary language, released with a KenLM n-gram decoder. Developed by the Language Technologies Unit, Bangor University. Output is lowercase without punctuation.

What changed in this release

A new model, trained from XLSR-53 on the Unit's 2607 data configuration. This is the first tagged release on this repository since 24.07 (July 2024). The Unit's wav2vec2 release in the meantime, wav2vec2-btb-cv-ft-cv-cy 24.10, went out under a separate name and is the comparison point in the tables below.

• Training data. 143 hours of Welsh speech (Banc Trawsgrifiadau Bangor 26.07, Corpws CLlLC / WLGA, Lleisiau ARFOR, Common Voice 23.0 and its VAD-segmented derivative) and 22 hours of UK- and Irish-accented English from Common Voice 23.0: the same corpus as the Unit's whisper-large-ft-cy-en 2607 and kaldi-cy 2607 releases, so the three model families can be compared on one benchmark. The 24.10 model was fine-tuned on Common Voice read speech only.
• Accuracy. Lower WER than 24.10 on every Welsh test set, both models decoded with their language models: by around 21 and 23 points on Banc Trawsgrifiadau Bangor and Lleisiau ARFOR (spontaneous speech), and by about one point on Common Voice and FLEURS (read speech), where 24.10, a Common Voice specialist, had led the previous Welsh-only releases. The mean over the four Welsh sets falls from 51.08 to 39.67.
• English. The model transcribes UK- and Irish-accented English as a secondary language; 24.10 is Welsh-only.
• n-gram decoder. Ships with a KenLM 5-gram decoder estimated on the training transcripts, used automatically when the model is loaded with Wav2Vec2ProcessorWithLM. On the benchmark it is worth 3.4 WER points over greedy decoding averaged across the five test sets (between 2 and 4 points per set), less than n-gram decoders usually give wav2vec2 models; the table under the results shows both.
• Licence. The first wav2vec2 release from the Unit to carry the Apache-2.0 LICENSE and NOTICE files described under Licence. The model may be used commercially.

Evaluation results

WER / CER (lower is better) on the held-out test sets, scored per utterance against the same references for every model compared.

The references are the test sets' transcripts as published, with their capitalisation and punctuation intact; only transcription-convention markers (such as <chwerthin>) are removed before scoring. This model emits lowercase text without punctuation, so every reference word that carries a capital letter or attached punctuation is scored as an error. The figures are therefore conservative, and higher than a normalised comparison would give, but every model in the table is scored the same way, so the comparison between them holds.

Decoding. The WER and CER columns are for this model decoded with its KenLM 5-gram decoder (revision 2607-20261021-kenlm), which is what the repository ships and what Wav2Vec2ProcessorWithLM loads. The greedy, no LM column is the same acoustic model decoded greedily (revision 2607-20261021). The 24.10 figures use that model's own KenLM decoder, which its published repository carries, so the two models are compared like for like.

Mean across the four Welsh test sets: WER 39.67, against 51.08 for 24.10: 11.41 points lower. Lower WER than 24.10 on every set.

Welsh transcription

English transcription

24.10 is techiaith/wav2vec2-btb-cv-ft-cv-cy, the Unit's previous wav2vec2 release: the Banc Trawsgrifiadau Bangor + Common Voice model fine-tuned further on Common Voice 18 with its encoder frozen. That made it a read-speech specialist, which is why it comes closest on the two read sets while trailing far behind on spontaneous speech. It is Welsh-only, so its figure on the English set measures nothing but that.

Training data

† Common Voice data is not redistributed here. techiaith/commonvoice_23_0_cy (Common Voice 23.0 Welsh), techiaith/commonvoice_vad_cy (Common Voice Welsh, VAD-segmented), techiaith/commonvoice_23_0_en__GB_IE (Common Voice 23.0 English, GB and IE accents) are held in private repositories, in line with the Mozilla Foundation's terms for obtaining and sharing Common Voice data. They are listed above for provenance and reproducibility, not as a download.

The Welsh and English Common Voice corpora are freely available from Mozilla themselves: Mozilla Common Voice datasets. Releases are distributed through the Mozilla Data Collective, and the datacollective package offers a programmatic download. With that in hand the rest of this pipeline is reproducible — every other dataset in the table is openly available on the Hugging Face Hub. Our thanks to Mozilla and to the speakers who contributed their voices.

As materialised for training, after curation and the duration filter: 142.8 hours of Welsh and 22.4 hours of English:

Training configuration

Language model (n-gram decoder)

The repository carries a KenLM 5-gram language model in language_model/, with the matching alphabet.json, for beam-search decoding with pyctcdecode. It is estimated on the transcripts of the training split only, decoded back from the labels the CTC head was trained on (so its vocabulary cannot contain a word the acoustic model is unable to produce; no validation or test transcript was used), 74,500 distinct words, with decoder weights alpha = 0.5 and beta = 1.5. Loading the model with Wav2Vec2ProcessorWithLM (below) uses it; loading with Wav2Vec2Processor decodes the acoustic model alone, greedily. The acoustic model is the same in both cases (training tag 2607-20261021).

Usage

pip install transformers torch librosa pyctcdecode kenlm

With the n-gram decoder (recommended):

import librosa
import torch
from transformers import Wav2Vec2ForCTC, Wav2Vec2ProcessorWithLM

processor = Wav2Vec2ProcessorWithLM.from_pretrained("techiaith/wav2vec2-xlsr-53-ft-cy-en", revision="26.10")
model = Wav2Vec2ForCTC.from_pretrained("techiaith/wav2vec2-xlsr-53-ft-cy-en", revision="26.10").eval()

audio, _ = librosa.load("welsh_audio.wav", sr=16_000)
inputs = processor(audio, sampling_rate=16_000, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits

print(processor.batch_decode(logits.numpy()).text[0])

Acoustic model only (greedy CTC decoding, no language model):

import librosa
import torch
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor

processor = Wav2Vec2Processor.from_pretrained("techiaith/wav2vec2-xlsr-53-ft-cy-en", revision="26.10")
model = Wav2Vec2ForCTC.from_pretrained("techiaith/wav2vec2-xlsr-53-ft-cy-en", revision="26.10").eval()

audio, _ = librosa.load("welsh_audio.wav", sr=16_000)
inputs = processor(audio, sampling_rate=16_000, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits

print(processor.batch_decode(torch.argmax(logits, dim=-1))[0])

The model was trained on clips of up to 20 seconds. Split longer recordings on silences first, or let the transformers pipeline chunk them:

from transformers import pipeline

pipe = pipeline("automatic-speech-recognition", model="techiaith/wav2vec2-xlsr-53-ft-cy-en", revision="26.10", chunk_length_s=20)
print(pipe("long_recording.wav")["text"])

Intended use and limitations

The model transcribes Welsh speech, and English speech with UK and Irish accents, to text. It is a research model, not a certified product, and is not a substitute for a qualified transcriber.

Automatic transcription makes mistakes. Output can be wrong while appearing fluent and confident. Known failure modes include omitted, inserted or substituted words; misspelt names and place names; numbers written as words or dropped, since the alphabet has no digits; English words in Welsh speech, and Welsh words in English speech, spelt as the other language, since the model does not identify the language being spoken; and reduced accuracy on dialect, overlapping speakers, noisy or distant audio, and subjects outside the training data. Output is lowercase and carries no punctuation. Treat output as provisional and have it checked and corrected by a competent person before it is relied on as a record, used in any decision affecting a person, or shared or published. Where the model is used interactively or in real time (live captions, dictation), tell users that what they see is an automatic transcription and may be wrong.

Not fit for medical, legal, financial, safety-critical or emergency settings, or any decision affecting a person's rights, health, liberty or livelihood, unless a qualified person reviews every output.

Use in Wales. The model was developed to help organisations provide services in Welsh that are as good as those they provide in English, not to replace Welsh-language provision. Under section 1 of the Welsh Language (Wales) Measure 2011 (2011 nawm 1), Welsh has official status in Wales, and that status is given legal effect by, among other things, the treatment of Welsh no less favourably than English (s. 1(2)(b)) and by standards of conduct relating to the use of Welsh (s. 1(3)(e)). These standards are made and enforced under Part 4 of the Measure. Bodies still operating a Welsh language scheme under Part II of the Welsh Language Act 1993 (c. 38) must give effect to the principle in section 5 of that Act that, in the conduct of public business and the administration of justice in Wales, Welsh and English should be treated on a basis of equality. Which duties apply depends on the body and the regulations or scheme that cover it, and responsibility for compliance rests with the body deploying the model. See the Welsh Language Commissioner's regulatory policy statement on artificial intelligence and the Welsh language, Technology, Websites and Software: Welsh Language Considerations, and the advice document Bilingual drafting, translation and using Welsh face to face. The legislation is at legislation.gov.uk/mwa/2011/1 and legislation.gov.uk/ukpga/1993/38.

Further limitations:

• Lowercase, no punctuation, no digits. The alphabet has no capitals, punctuation or numerals; numbers come out as words or not at all.
• Clips, not recordings. Trained on 1–20 second clips; longer audio must be chunked (see Usage).
• English is secondary. Fine-tuned on 22 hours of read UK- and Irish-accented English beside 143 hours of Welsh, it is far weaker than a dedicated English model. It does not identify the language being spoken: one alphabet and one n-gram model serve both, so English words in Welsh speech, and Welsh ones in English, may be spelt the other way.
• Domain of the decoder. The n-gram model is estimated on the training transcripts only, so decoding is biased towards their vocabulary and registers; the acoustic model can still spell out a word the decoder has never seen, at a penalty.
• The validation WER in trainer_state.json is not a result. It selected the checkpoint; the Common Voice half of that validation split overlaps the training slice, so read the model's accuracy off the benchmark tables above.

Licence

Released under the Apache License 2.0. The accompanying NOTICE file carries the University's attribution, the warning that automatic transcription makes mistakes, and the attributions required by the training-data and base-model licences. Section 4 of the licence requires anyone who redistributes the model or a derivative to keep that NOTICE.

One-line attribution: Includes wav2vec2-xlsr-53-ft-cy-2607, developed by the Language Technologies Unit, Bangor University, and released under the Apache License 2.0.

Acknowledgements

Built on XLSR-53, pre-trained and released by Meta AI under the Apache License 2.0.

Developed by Uned Technolegau Iaith, Prifysgol Bangor / Language Technologies Unit, Bangor University.

Funded by the Welsh Government.