KazBERT-benchmark

KazBERT Benchmark 🇰🇿 — Kazakh Encoders Compared

🤖 Fully AI-generated. Design, code (benchmark.py, benchmark_unt.py), plots and this card were produced end-to-end by an AI agent (Claude, Hermes ML research loop). Compute: a single Kaggle T4, inference-only, no training. No credentials used or embedded — all models/datasets are public and the code is included for transparency.

ModernBERT-style comparison of KazBERT vs four Kazakh-capable encoders, zero-shot, across tokenizer efficiency, exam-QA accuracy and embedding quality.

TL;DR

• ⚡ Smallest & fastest: KazBERT hits top-tier quality with a 32k vocab (vs 120k–250k) and the fastest inference.
• 🎯 Beats multilingual on Kazakh: on tokenizer efficiency and MLM answer-scoring (PLL), KazBERT and the Kazakh-specific models clearly outperform mBERT and XLM-R.
• 📊 Two evaluations: (1) dastur-mc reading MC, (2) the full Kazakh ЕНТ / UNT — 14,850 real exam questions across 7 subjects.
• 🥈 Honest: KazBERT isn't #1 on every metric, but it's consistently in the leading group at the smallest footprint.

---

Part 1 — dastur-mc (reading comprehension MC) + tokenizer

(bold = best in column; random MC = 25%)

• Fertility — subword tokens per Kazakh word (lower = better tokenizer). Fair across models.
• MC-PLL — zero-shot MC where each option is scored by length-normalised pseudo-log-likelihood under the MLM head. Main task signal.
• MC-embed — same task via cosine of mean-pooled embeddings (diagnostic; all these encoders are MLM-only, so raw embeddings are weak for everyone).

Qualitative fill-mask (illustrative, not scored)

Астана — Қазақстанның [MASK] қаласы.

Мен қазақ [MASK] сөйлеймін.

Абай Құнанбаев — ұлы қазақ [MASK].

KazBERT and the Kazakh-trained models return fluent Kazakh; mBERT often falls back to sub-word fragments or [UNK].

---

Part 2 — Full Kazakh ЕНТ / UNT (14,850 questions, 7 subjects)

Zero-shot on every question of kazakh-unified-national-testing-mc (4–8 options each). Random baseline ≈ 17.9%.

Per-subject (PLL accuracy)

---

⚠️ Caveats

• Zero-shot, no fine-tuning — intrinsic probes, not fine-tuned scores; fine-tuning could reorder models.
• Cross-tokenizer comparisons are imperfect — PLL/fill-mask depend on how each vocab segments text; numbers are indicative.
• MLM answer-scoring ≠ reasoning — exams need world knowledge these small encoders lack, so absolute accuracies are modest; the point is the relative Kazakh-language signal.

Reproduce

Runs on a free Kaggle T4 with pip install transformers datasets — public models/datasets only:

python benchmark.py       # Part 1: dastur-mc + tokenizer
python benchmark_unt.py   # Part 2: full ЕНТ

Provenance

• Models: KazBERT, mBERT, XLM-R base, kaz-roberta-conversational, KazakhBERTmulti
• Data: kazakh-unified-national-testing-mc, kazakh-dastur-mc, kazakh_wiki_articles
• Compute: Kaggle Notebook, 1× NVIDIA Tesla T4 · Pipeline: Hermes ml-research-loop
• Author: AI agent (Claude). Human: chose the models and pressed go.