lma-indic-models-shrey715

LMA Indic Models -- Hindi & Nepali Decoder-Only Transformer LMs

Two fully independent, from-scratch (no pretrained models, no pretrained tokenizers, no HuggingFace transformers classes) decoder-only Transformer language models -- Model H (Hindi, higher-resource) and Model L (Nepali, lower-resource) -- built across three phases: Phase 1 (data pipeline + BPE tokenizer), Phase 2 (pretraining + evaluation, incl. a bonus no-positional-embedding ablation), Phase 3 (synthetic reasoning finetuning + attention analysis). 26,524,160 parameters each, identical architecture, entirely separate data/vocabulary/weights between languages. Generated by scripts/publish/push_to_hf.py (data) and scripts/publish/push_checkpoints_to_hf.py (checkpoints, under checkpoints/); see the source repository's report/phase{1,2,3}/report.tex for full methodology.

Phase 1 -- Data & Tokenizer

Hindi

• Corpus: not available
• Tokenizer: chosen vocab=8,192 (all 5 swept sizes included in this repo: 4,096, 8,192, 16,384, 32,768, 65,536), fertility=1.390 tokens/word, UNK rate=0.0000%, byte-fallback rate=0.0000%
• Perplexity filter: 5-gram KenLM, reject threshold=4768.99 perplexity

Nepali

• Corpus: not available
• Tokenizer: chosen vocab=8,192 (all 5 swept sizes included in this repo: 4,096, 8,192, 16,384, 32,768, 65,536), fertility=1.530 tokens/word, UNK rate=0.0000%, byte-fallback rate=0.0000%
• Perplexity filter: 5-gram KenLM, reject threshold=16796.45 perplexity

Phase 2 -- Pretraining

• Hindi: val ppl=20.970 bpb=0.4715, test ppl=20.504 bpb=0.4656 -- checkpoint: checkpoints/hindi/latest.pt
• Nepali: val ppl=23.273 bpb=0.4101, test ppl=23.463 bpb=0.4133 -- checkpoint: checkpoints/nepali/latest.pt

Full per-variant metrics (main + NoPE ablation + finetuned) and loading instructions: checkpoints/README.md.

Phase 3 -- Reasoning Finetuning

Hindi

• Synthetic reasoning dataset: 12,600/2,700/4,500 train/val/test examples, 6 attribute domains x 6 template families (incl. a held-out chain_5hop structural-generalization test), leakage audit: passed
• Pretrained vs. finetuned: exact-match pretrained=0.000 -> finetuned=0.334 -- checkpoint: checkpoints/hindi_finetune/latest.pt

Nepali

• Synthetic reasoning dataset: 12,600/2,700/4,500 train/val/test examples, 6 attribute domains x 6 template families (incl. a held-out chain_5hop structural-generalization test), leakage audit: passed
• Pretrained vs. finetuned: exact-match pretrained=0.000 -> finetuned=0.208 -- checkpoint: checkpoints/nepali_finetune/latest.pt

Structure

hindi/
├── tokenizer/                    # all 5 swept BPE vocab sizes (4,096, 8,192, 16,384, 32,768, 65,536), .model+.vocab each
├── perplexity_filter/            # KenLM 5-gram fluency filter used during cleaning
├── reasoning/{train,val,test}.jsonl.gz + generation_stats.json  # Phase 3 synthetic reasoning dataset
└── data/
    ├── processed/{train,val,test}/*.jsonl.gz  # final pretraining corpus splits
    └── raw/
        ├── downloaded/           # Sangraha subset actually used (gzipped)
        └── manual/                # self-collected news/Wikipedia/archive.org (gzipped)
nepali/
├── tokenizer/                    # all 5 swept BPE vocab sizes (4,096, 8,192, 16,384, 32,768, 65,536), .model+.vocab each
├── perplexity_filter/            # KenLM 5-gram fluency filter used during cleaning
├── reasoning/{train,val,test}.jsonl.gz + generation_stats.json  # Phase 3 synthetic reasoning dataset
└── data/
    ├── processed/{train,val,test}/*.jsonl.gz  # final pretraining corpus splits
    └── raw/
        ├── downloaded/           # Sangraha subset actually used (gzipped)
        └── manual/                # self-collected news/Wikipedia/archive.org (gzipped)
checkpoints/<lang>{,_no_pos_emb,_finetune}/latest.pt  # pushed separately, see checkpoints/README.md

.crawl_state_*.json scraper resume-checkpoint files are deliberately excluded (internal bookkeeping, not corpus content).

Loading the corpus

datasets.load_dataset handles gzip-compressed JSONL natively -- no manual decompression needed:

from datasets import load_dataset
ds = load_dataset("json", data_files="hindi/data/processed/train/train.jsonl.gz", split="train")

Loading the tokenizer

All 5 swept vocab sizes are included per language (4,096, 8,192, 16,384, 32,768, 65,536); this project currently uses 8,192 for both (see report/phase1/report.tex, Vocabulary Size Selection, for why):

import sentencepiece as spm
sp = spm.SentencePieceProcessor(model_file="hindi/tokenizer/hindi_bpe_8192.model")

Loading the perplexity filter

import kenlm
model = kenlm.Model("hindi/perplexity_filter/perplexity_filter.klm")

Loading the reasoning dataset

from datasets import load_dataset
ds = load_dataset("json", data_files="hindi/reasoning/train.jsonl.gz", split="train")

Loading a checkpoint

Not HuggingFace-transformers-compatible; load with the project's own scripts.model.transformer.MonolingualDecoderLM (see checkpoints/README.md for the full snippet and per-variant metrics).