lma_models

LMA — Hindi and Nepali tokenizers and language models

This repository holds both phases of the project: the two sentencepiece tokenizers (Phase 1) and the two pretrained decoder-only Transformers that use them (Phase 2).

LMA Phase 1 — Hindi and Nepali tokenizers

Two independent sentencepiece tokenizers, one per language, trained from scratch for a pair of ~25M-parameter decoder-only Transformers.

They share nothing — not merges, not pieces, not a vocabulary file. Hindi and Nepali both use the Devanagari block (U+0900–U+097F), so keeping the two corpora and the two vocabularies separate is the central constraint of the project rather than an afterthought.

How these were chosen

Twenty models were compared: five vocabulary sizes (8k, 10k, 12k, 14k, 16k) across two algorithms (BPE, unigram), for both languages. Every model read the same 10% random sample of its language's training split, so differences between them are differences between models rather than between samples.

Unigram beat BPE at every vocabulary size in both languages — ten paired comparisons, no exceptions, by 1.2–2.7% fertility.

Vocabulary 10,000 was selected over 16,000 despite 16k tokenizing better. The embedding matrix is vocab_size × 512 parameters against a 25M budget, so 16k spends 33% of the whole model on a lookup table while 10k spends 20%. The selection rule is the smallest vocabulary whose fertility is within 8% of the best, which trades ~5–7% fertility for ~3.1M parameters returned to the transformer layers.

Hindi — fertility 1.3250 tokens/word, 3.7749 chars/token, 0 UNK, 0.289% byte-fallback, 201 unused pieces. Nepali — fertility 1.4620 tokens/word, 4.5208 chars/token, 0 UNK, 0.178% byte-fallback, 239 unused pieces.

UNK is impossible. byte_fallback=True decomposes any unseen character into byte tokens, so the UNK count is zero by construction rather than by luck.

Training data

Splits are document-level, stratified by source, 70/15/15 by characters, with one whole source held out per language as an unseen-domain test set. The tokenizers saw the training split only.

The corpora are at meet5568/lma_datasets.

Usage

import sentencepiece as spm
from huggingface_hub import hf_hub_download

path = hf_hub_download("meet5568/lma_models", "tokenizer/hindi/hi_tokenizer.model")
sp = spm.SentencePieceProcessor(model_file=path)
print(sp.encode("भारत एक विशाल देश है।", out_type=str))

Limitations

Unigram's memory scales with total corpus length — it builds a suffix array over every character — at roughly 10.6 GB of RAM per GB of text, measured. The full training split would need about 49 GB, so the final unigram models were trained on 50% of it. A control experiment found fertility differing in the fourth decimal place between 81.6% and 100% of the lines, so this is a hardware limit rather than a quality one, but it is stated rather than implied.

---

LMA Phase 2 — the pretrained models

Two decoder-only Transformers, one per language, pretrained from scratch on the corpora above. They share an implementation and nothing else: separate data, separate tokenizers, separate vocabularies, separate weights.

Written from PyTorch primitives only — nn.Linear, nn.Embedding, nn.LayerNorm, nn.Dropout. No nn.Transformer*, no nn.MultiheadAttention, no F.scaled_dot_product_attention; the attention is written out.

Results

Compare the two models on bits per byte, not perplexity. Perplexity is measured per token and the two tokenizers differ (10.10 bytes per token for Hindi against 11.94 for Nepali), so Nepali is charged for packing more text into each prediction. Bits per byte divides by the raw UTF-8 length and reverses the ranking. The same caution applies to comparing these numbers with a model trained on a different corpus with a different tokenizer.

How the architecture was chosen

A full factorial sweep per language — learning rate x dropout x shape x context = 24 trials each, 48 in total, every trial 400 steps at 26.2M tokens. All 48 completed and none diverged. Both languages independently chose the same configuration. Learning rate mattered most (1.2e-3 beat 6e-4 everywhere); width beat depth at a fixed parameter budget; context 512 beat 1024, because documents average 82-128 tokens; dropout 0.0 beat 0.1, as a single epoch has nothing to overfit.

Training

AdamW (betas 0.9/0.95, weight decay 0.1 on matrices only), 400-step warmup then cosine from 0.0012 to 1.2e-04, 65,536 tokens per optimisation step, gradient clipping 1.0, AMP fp16. One epoch, so every token is seen exactly once.

Usage

import torch
from huggingface_hub import hf_hub_download

path = hf_hub_download("meet5568/lma_models", "pretrained/hindi/best.pt")

# weights_only=False is required: the checkpoint carries optimizer, scaler and
# RNG state so a run can resume exactly, and torch 2.6+ refuses those under the
# default weights_only=True.
ckpt = torch.load(path, map_location="cpu", weights_only=False)

print(ckpt["model_config"])                 # architecture
print(ckpt["step"], ckpt["best_val_loss"])  # where it stopped
state = ckpt["model"]                       # the weights themselves

Each checkpoint holds the model weights, optimizer state, gradient-scaler state, the training step, the epoch and window cursor, and both configs — so a run resumes exactly where it stopped rather than approximately.

run.json beside each checkpoint repeats the architecture and the training hyperparameters in plain text and adds the full loss curve, so the settings can be read without loading 317 MB of weights.

The model code is not on the hub. ckpt["model"] is a plain state_dict for the implementation in the project repository, whose common/model_train/model/ builds the architecture from model_config.

Limitations

Both models are under-trained. At ~26.4M parameters the Chinchilla-optimal budget is roughly 528M tokens; one epoch gives 17.4 tokens per parameter for Hindi and 14.9 for Nepali, against the ~20 rule of thumb. Both losses were still falling when the epoch ended.

The sweep ranked 400-step behaviour, and its winner sits on the boundary of the grid on three of four axes, so those values are the best tested rather than the best. One seed per configuration.

The unseen-domain split did not work as a robustness test — both models score better on it than in domain by perplexity, which says that domain is simpler than the corpus average rather than that the models generalise well.

---

LMA Phase 3 — the reasoning-finetuned models

The Phase 2 checkpoints above, finetuned on a synthetic comparative reasoning corpus generated separately for each language, at five sample sizes (5k, 10k, 15k, 30k, 50k, 100k). Every size is an independent run from the pretrained checkpoint: 5 epochs, batch 32, peak learning rate 1e-4 with warmup and cosine decay, loss on the answer tokens only, best validation checkpoint kept. Same architecture, tokenizer and 26.4M parameters as Phase 2.

The data

Four entity domains (people, cities, items, vehicles), ten attributes (age, height, score, weight, income, experience, temperature, population, price, speed), four families (compare, transitive, superlative, ordering), six phrasings each — every word counted in the pretraining corpus first. 100,000 training examples; smaller sizes are prefixes. Corpora: meet5568/lma_datasets under reasoning/.

600 items per split; 0 duplicate prompts across all splits.

Results

Each cell is Choice / Gen (Invalid): choice accuracy ranks the legal answers by log-probability; generation accuracy is greedy decoding, first word (punctuation stripped) exact; invalid is the share of generations that are not a legal answer. PPL and BPB are measured on the held-out pretraining corpus.

Hindi (Model H)

Nepali (Model L)

What the tables show

Hindi. Pretrained, the model is close to chance (36.8% choice against 37.4%) and produces a legal answer in generation 0.2% of the time.

• The answer format is learned at every size. On the control split D the invalid rate falls from 99.3% to 0.0%–1.8% across 5k–100k.
• Learning appears at 100k. Test D choice accuracy 5k 40.5%, 10k 39.3%, 15k 39.5%, 30k 37.8%, 50k 40.0%, 100k 64.8%, against a majority baseline of 47.0%.
• At 100k: A 47.0%, B 47.7%, C 52.2%, D 64.8%, E 79.7% choice accuracy; majority baseline 44.5% / 44.5% / 44.3% / 47.0% / 54.8%.
• Unseen names are the costliest axis. At 100k, unseen values cost 4.5 points (C − B), unseen names 12.7 (D − C), and unseen templates -0.7 (A − B).
• Longer chains (E). 3–4-fact chains score 79.7% against 92.0% on D's 2-fact transitive items (-12.3 points), with 54.8% for always giving E's commonest answer.
• Generation of names. At 100k, superlative items — whose answer is an entity name — are invalid 97.3% of the time on split A (held-out names) against 3.3% on D (training names): the model writes a name it saw in training rather than the new name in the prompt.
• Finetuning costs language modelling from the first row. Corpus perplexity rises 1.42x at 5k (25.66 → 36.42) and 67.06x at 100k (1720.75); bits-per-byte 0.4637 → 1.0647.

Nepali. Pretrained, the model is close to chance (39.1% choice against 40.0%) and produces a legal answer in generation 0.0% of the time.

• The answer format is learned at every size. On the control split D the invalid rate falls from 100.0% to 3.8%–20.8% across 5k–100k.
• Learning appears at 100k. Test D choice accuracy 5k 41.7%, 10k 38.2%, 15k 40.2%, 30k 43.0%, 50k 42.5%, 100k 58.0%, against a majority baseline of 44.3%.
• At 100k: A 47.0%, B 45.5%, C 49.2%, D 58.0%, E 50.3% choice accuracy; majority baseline 44.5% / 44.5% / 44.3% / 44.3% / 51.5%.
• Unseen names are the costliest axis. At 100k, unseen values cost 3.7 points (C − B), unseen names 8.8 (D − C), and unseen templates +1.5 (A − B).
• Longer chains (E): not interpretable here. Even D's 2-fact transitive items stay at 46.7% (chance 50.0%), so this model never learned to chain two facts and E (50.3%) cannot show a chain-length effect; its gains come from the other question types.
• Generation of names. At 100k, superlative items — whose answer is an entity name — are invalid 100.0% of the time on split A (held-out names) against 14.7% on D (training names): the model writes a name it saw in training rather than the new name in the prompt.
• Finetuning costs language modelling from the first row. Corpus perplexity rises 1.56x at 5k (37.38 → 58.26) and 638.57x at 100k (23869.70); bits-per-byte 0.4374 → 1.2177.

Usage

import torch
from huggingface_hub import hf_hub_download

path = hf_hub_download("meet5568/lma_models", "finetuned/hindi/100k/best.pt")
ckpt = torch.load(path, map_location="cpu", weights_only=False)

Prompt with the question followed by the answer marker (जवाब for Hindi, जवाफ for Nepali) and read the next word.

Limitations

The task is synthetic and templated. Finetuning on it degrades general language modelling, visible in the PPL column, and the models almost never generate a held-out entity name.