swahili-5gram-lm

Swahili 5-gram Language Model

A KenLM language model trained on 20969855 Swahili sentences from Common Voice.

Usage with pyctcdecode

from pyctcdecode import build_ctcdecoder
from huggingface_hub import hf_hub_download
import json

# Download files
vocab_path = hf_hub_download("Godsonntungi2/swahili-5gram-lm", "vocab.json")
lm_path = hf_hub_download("Godsonntungi2/swahili-5gram-lm", "5gram.arpa")
unigrams_path = hf_hub_download("Godsonntungi2/swahili-5gram-lm", "unigrams.txt")

# Load vocab
with open(vocab_path) as f:
    vocab_dict = json.load(f)
vocab_list = [k for k, v in sorted(vocab_dict.items(), key=lambda x: x[1])]

# Load unigrams
with open(unigrams_path) as f:
    unigrams = [line.strip() for line in f]

# Build decoder
decoder = build_ctcdecoder(
    labels=vocab_list,
    kenlm_model_path=lm_path,
    unigrams=unigrams,
)

# Decode with LM
text = decoder.decode(logits)

Training Data

• Source: Common Voice 19.0 Swahili
• Sentences: 20969855
• Vocabulary: 50 characters

Files

• 5gram.arpa - ARPA format LM
• vocab.json - Character vocabulary
• unigrams.txt - Word unigrams for boosting
• corpus.txt - Training text corpus