Downloads · 30 days
14
6% of all-time downloads
janmbuys/omniASR-CTC-300m-v2-Zulu-Lwazi
omniASR-CTC-300m-v2-Zulu-Lwazi is a automatic speech recognition model from janmbuys. Use it when you need speech turned into text. It is set up for transformers. The card lists the license as cc-by-4.0.
A standalone Wav2Vec2ForCTC model: the uctnlp/omniASR-CTC-300m-v2-Zulu-Baseline checkpoint with a LoRA adaptation to 8 kHz telephone speech already merged in, plus a 6-gram subword language model and the lexicon neede…
Downloads · 30 days
14
6% of all-time downloads
All-time downloads
245
Public
Parameters
326M
2.7 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors1.3 GB · 50%
From the Hugging Face model README
A standalone Wav2Vec2ForCTC model: the
uctnlp/omniASR-CTC-300m-v2-Zulu-Baseline
checkpoint with a LoRA adaptation to 8 kHz telephone speech already merged in,
plus a 6-gram subword language model and the lexicon needed to decode with it.
Loads with plain from_pretrained — no peft, no separate base download:
from transformers import Wav2Vec2ForCTC, Wav2Vec2CTCTokenizer
model = Wav2Vec2ForCTC.from_pretrained("<repo-id>")
tok = Wav2Vec2CTCTokenizer.from_pretrained("<repo-id>")
The LoRA adapter is also kept under lora/ for provenance, should you want to
re-merge it onto a different base.
| Model | Wav2Vec2ForCTC, 326M params, character-level CTC (10,288 classes) |
| Adaptation | LoRA r=16, α=32 on attention + FFN (7.1M params, 2.1%), merged |
| Adapted on | Lwazi isiZulu train split, 4,562 utterances (~7 h) |
| LM | 6-gram over SentencePiece BPE-32k subwords, KenLM |
| LM data | MzansiText isiZulu 71.6M words + English 5.9M (7.6%) |
Lwazi isiZulu, scored against fragment-resolved references
(lm_weight=0.75, word_score=-0.5, beam 100):
| system | validation WER / CER |
|---|---|
| base, greedy | 53.36 / 18.58 |
| base + LM | 46.05 / 18.57 |
| + LoRA, greedy | 38.16 / 9.44 |
| + LoRA + LM | 31.58 / 8.78 |
For reference, the same LM on 48 kHz studio speech (African Next Voices isiZulu dev, 3,063 utts) gives 21.16% → 17.10% WER. Lwazi is much harder: narrowband telephone audio, spontaneous speech, ~10% OOV.
tokenizer_config.json in the base checkpoint declares
word_delimiter_token: "▁", but ▁ is not in vocab.json. The real word
delimiter is a literal space, id 4.
Decoding with the shipped setting happens to work — no ▁ is ever emitted,
so the replacement is a no-op. But encoding maps every space to <unk>
(id 3). Fine-tuning with that tokenizer trains the model to emit <unk> at
every word boundary: the loss falls steadily while WER climbs past 100%.
This repo ships a corrected tokenizer_config.json, so
Wav2Vec2CTCTokenizer.from_pretrained("<repo-id>") is already right:
tok = Wav2Vec2CTCTokenizer.from_pretrained("<repo-id>")
assert tok("a b").input_ids == [3859, 4, 3425] # 4 is the space, not <unk>
If you point at the original base checkpoint instead, override it explicitly:
tok = Wav2Vec2CTCTokenizer("vocab.json", unk_token="<unk>", pad_token="<s>",
bos_token="<s>", eos_token="</s>",
word_delimiter_token=" ") # NOT "▁"
Despite the base model card's wording, this checkpoint emits characters:
all 10,284 non-special entries in vocab.json are single characters. The BPE
subwords here are language model units, not acoustic units. CTC blank is
pad_token_id = 0.
audio -> CTC -> character posteriors -> flashlight LexiconDecoder -> text
| |
lexicon | | KenLM (6-gram)
subword -> characters over subwords
The lexicon bridges the two. Word boundaries ride on a leading space:
▁ngi -> | n g i word-initial: consumes the preceding space
ya -> y a word-medial: does not
- -> -
Consequences to respect:
sil to blank, not to space. Boundaries already come from the
leading-space spellings; also treating space as optional silence
double-counts them and measurably hurts (with a positive word_score it
degenerates badly).lm_weight ≈ 0.75 is optimal, well below the 1.5–2.5 typical of
word-level lexicon decoding, because a subword LM fires several times more
often per utterance. This value was optimal on both studio and telephone
speech.keep_ids.npy selects them; slice the
posteriors and renormalise. This is ~250× smaller and much faster.model.safetensors merged model (base + LoRA), 1.3 GB fp32
config.json Wav2Vec2ForCTC config
vocab.json 10,288 character classes
tokenizer_config.json CORRECTED: word_delimiter_token is " "
preprocessor_config.json 16 kHz, do_normalize
special_tokens_map.json
lora/ LoRA adapter, for provenance / re-merging
lm/lm6.bin KenLM 6-gram over BPE-32k subwords (binary trie)
lm/lexicon.txt subword -> character spellings
lm/tokens.txt 39-token reduced set ('#' = blank, '|' = space)
lm/keep_ids.npy column indices into the 10,288-class output
lm/bpe32000.model SentencePiece model (to re-encode text for LM training)
inference_example.py
Greedy decoding needs only transformers:
import torch, soundfile as sf
from transformers import Wav2Vec2ForCTC, Wav2Vec2CTCTokenizer
model = Wav2Vec2ForCTC.from_pretrained("<repo-id>").eval()
tok = Wav2Vec2CTCTokenizer.from_pretrained("<repo-id>")
wav, sr = sf.read("utt.wav", dtype="float32") # resample to 16 kHz first
wav = (wav - wav.mean()) / (wav.std() + 1e-7)
with torch.no_grad():
ids = model(torch.from_numpy(wav)[None]).logits.argmax(-1)[0]
print(tok.decode(ids.tolist()))
LM decoding additionally needs flashlight-text built with KenLM; the PyPI wheel is not:
pip install torch torchaudio transformers soundfile numpy sentencepiece kenlm
USE_KENLM=1 CMAKE_POLICY_VERSION_MINIMUM=3.5 \
pip install --no-build-isolation --no-binary flashlight-text flashlight-text
KenLM then lives at flashlight.lib.text.decoder.kenlm.KenLM — a submodule, not
re-exported into flashlight.lib.text.decoder. On Anaconda you may also need
conda install -c conda-forge "libstdcxx-ng>=13", since the bundled
libstdc++ (3.4.29) is too old for the kenlm wheel.
# from a local clone of this repo
python3 inference_example.py --audio utt.wav # LM decoding
python3 inference_example.py --audio utt.wav --greedy # no LM
Batch size 1 with gradient accumulation 8, lr 1e-4 (OneCycle), 8 epochs, best
epoch by validation WER. Batch 1 is deliberate: the feature extractor sets
return_attention_mask=False, so the model was trained without an attention
mask, and padding a batch would push unmasked silence through a
stable-layer-norm encoder.
The adapter was trained on fragment-resolved transcriptions. Lwazi marks
false starts, bound concord prefixes and truncations all with a trailing
hyphen; naive cleaning turns elaw- elawini into elaw elawini and ku- Peter
into ku peter. Training on resolved text instead was worth 3.9 WER points on
test.
uctnlp/omniASR-CTC-300m-v2-Zulu-Baseline