gemma4-sinhala-cpt

Gemma-4-E4B Sinhala CPT

Continually pre-trained variant of google/gemma-4-E4B adapted for Sinhala using a Grapheme Pair Encoding (GPE) tokenizer extension and LoRA-based continual pre-training on the MADLAD_CulturaX_cleaned Sinhala corpus.

This model is part of ongoing research on knowledge-graph-driven hybrid RAG for Sinhala institutional information retrieval (university admissions domain).

Tokenizer extension

Sinhala is an abugida script where standard byte/codepoint-level BPE tokenization fragments grapheme clusters (virama-bound consonant conjuncts) into linguistically meaningless pieces. This model extends Gemma-4-E4B's tokenizer using Grapheme Pair Encoding (GPE), following Velayuthan & Sarveswaran (2024), implemented via the SLTK library.

• GPE vocabulary trained on 500,000 sentences (58.2M characters) from polyglots/MADLAD_CulturaX_cleaned
• 4467 new grapheme-cluster tokens added to Gemma's vocabulary
• New vocabulary size: 266611
• New token embeddings mean-initialized from constituent byte-level sub-token embeddings (not randomly initialized)

Training procedure

1. Embedding warm-up (3,000 steps): transformer frozen, only new GPE embedding rows trained, LR=1e-3
2. Continual pre-training (50,000 steps): LoRA (r=128, alpha=256) on all attention and FFN layers, LR=1e-4 cosine schedule, block_size=512, causal language modeling objective on Sinhala text

Evaluation

Held-out evaluation set: 500 Sinhala sentences from MADLAD_CulturaX_cleaned, excluded from the CPT training range.

Embedding drift: mean cosine similarity between new-token embeddings before and after CPT: 1.0 — confirms the new GPE embeddings moved substantially during training rather than remaining at their mean-initialization starting point.

Note: GPE-only perplexity is expected to be very high — this measures the model immediately after vocabulary extension and embedding resize, before any training has occurred on the new tokens. CPT recovers this to a usable range, which is the primary validation of the continual pre-training stage.

Intended use

This is a continually pre-trained base model intended as a starting point for supervised fine-tuning (SFT) on downstream Sinhala NLP tasks. It has not been instruction-tuned. For the SFT-adapted version specialized for Subject-Predicate-Object (SPO) triple extraction, see the related repository.

Limitations

• Trained primarily on general web text (CulturaX); domain-specific institutional vocabulary (university admissions terminology) is only partially covered
• Not instruction-tuned — not suitable for direct chat/instruction use
• Evaluation limited to perplexity, fertility, and qualitative generation samples; downstream task performance not yet measured

Citation

If you use this model, please cite the GPE tokenization approach:

Velayuthan, M., & Sarveswaran, K. (2024). Egalitarian Language Representation in Language Models: It All Begins with Tokenizers.