KenLM-V1

KenLM-V1 — six-language decoding bundle

This repository publishes the frozen language-model assets used by the Afri Voices ASR experiments for Swahili, Kikuyu, Kalenjin, Dholuo, Somali and Maasai.

Profiles

• profiles/best_public_036529.json: historical decoding profile associated with the best known public leaderboard macro-WER 0.36529. That historical pipeline also used specialist acoustic checkpoints; this profile alone cannot reproduce 0.36529 and is not the default for the published single acoustic model.
• profiles/current_model_036878.json: exact decoding profile compatible with VynoDePal/omniASR-fine-tuning-6languageKenya, whose complete public submission scored 0.36878.
• active_profile.json records both roles explicitly.

Assets

Seven content-unique binary/unigram pairs are included because the two profiles share most assets while Somali uses an additional historical model in one route.

Each profile defines 12 routes (language|scripted and language|unscripted) with exact alpha, beta and beam width. Assets are content-addressed and deduplicated by SHA256.

Usage

import importlib.util
from pathlib import Path
from huggingface_hub import snapshot_download
repo = snapshot_download("VynoDePal/KenLM-V1", revision="v1.0.0")
spec = importlib.util.spec_from_file_location("kenlm_v1", Path(repo) / "load_decoder.py")
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
load_decoder = module.load_decoder

# labels must be the exact omniASR CTC label list, with blank at index 0.
decoder, beam, route = load_decoder(
    repo, labels, language="kln", domain="unscripted",
    profile="current_model_036878",
)
hypothesis = decoder.decode(log_probs, beam_width=beam)

Important compatibility rules

• Use the exact CTC tokenizer/labels from the acoustic model.
• Select the correct language and scripted/spontaneous domain.
• Keep only one decoder resident at a time for the edge-memory target.
• Apply the same Unicode/text normalization used by the ASR pipeline.
• These LMs are not language-identification or domain-classification models.

Evaluation and limitations

Leaderboard WER cannot isolate KenLM gains because acoustic checkpoints and decoding settings are coupled. Local held-out audits were used for selection, and some groups prefer historical settings. Language models can hallucinate frequent phrases or suppress rare names and code-switching.

Data and license

No raw training corpus or complete transcript collection is included. Derived unigram vocabularies and compiled n-gram weights are included and may preserve words or names observed in training. See ATTRIBUTION.md and license_audit.json; verify upstream terms for your intended use.

Future V6 update

The candidate produced by 20.K3/20.K4 is not included until it passes acoustic WER audit. A validated update will be committed to this same repository and tagged as a new version; tag v1.0.0 will remain immutable.