Gemma3-Singlish-Sinhala-CodeMix

Gemma3-Singlish-Sinhala-CodeMix

A fine-tuned Gemma-3-270M-IT model for Singlish -> Sinhala transliteration and code-mixed text translation.

The model handles both:

• Pure Singlish: Roman-script Sinhala → Sinhala Unicode
• Code-mixed text: English-Sinhala mixed text → pure Sinhala (English words translated to Sinhala)

Examples

Evaluation Results

Phonetic Test Set

Adhoc Test Set

Sample Predictions

Phonetic Test

Adhoc Test

Code-Mix Translation

Training

Architecture & Method

• Base model: google/gemma-3-270m-it (273M parameters)
• Method: Multi-phase LoRA fine-tuning
• Saved precision: bfloat16

Phase 1–3: Transliteration Foundation

The base transliteration model (Gemma3-Singlish-Sinhala-Merged) was trained in 3 progressive phases on the Swa-bhasha phonetic dataset:

Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

After each phase, the LoRA adapter was merged into the base model.

Phase 4: Code-Mix Fine-tuning

A new LoRA adapter was trained on top of the merged Phase 3 model to add code-mix translation capability:

Anti-Forgetting Strategy

• Replay buffer: 13K transliteration samples mixed into code-mix training
• Small LoRA rank (r=32): limits capacity to prevent overwriting base knowledge
• Low learning rate (3e-5): gentle updates to existing weights

Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
import warnings, os
warnings.filterwarnings("ignore")

MODEL_ID = "savinugunarathna/Gemma3-Singlish-Sinhala-CodeMix"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, torch_dtype=torch.bfloat16,
    trust_remote_code=True,
).to("cuda" if torch.cuda.is_available() else "cpu")
model.eval()

PROMPT = "Translate the following code-mixed text into pure Sinhala:\n{input}\nSinhala:"

gen_config = GenerationConfig(
    max_new_tokens=256, num_beams=3, do_sample=False,
    repetition_penalty=1.2, pad_token_id=tokenizer.pad_token_id,
    eos_token_id=tokenizer.eos_token_id, top_p=None, top_k=None,
)

def convert(text):
    prompt = PROMPT.format(input=text)
    inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=True).to(model.device)
    in_len = inputs["input_ids"].shape[1]
    with torch.no_grad():
        out = model.generate(**inputs, generation_config=gen_config)
    return tokenizer.decode(out[0, in_len:], skip_special_tokens=True).strip()

# Transliteration
print(convert("mama gedara yanawa"))            

# Code-mix translation
print(convert("ara politician wa mama dannawa"))  
print(convert("meka harima boring panthiyak"))   
print(convert("oyage phone eka ko"))              

Prompt Format

Single unified prompt for both tasks:

"Transliterate the following Singlish text to Sinhala by considering the context:\n{input}\nSinhala:"

Limitations

• Conjunct consonants: Occasionally produces ්ය instead of ්‍ය (ZWJ variants)
• Punctuation: May drop or alter commas and periods
• Rare English words: Very uncommon English words in code-mix may be transliterated phonetically instead of translated
• Long sequences: Quality may degrade for inputs longer than ~200 tokens

Model Details

Citation

If you use this model, please cite:

@misc{gunarathna2025codemix,
  title={Gemma3-Singlish-Sinhala-CodeMix: Multi-Phase LoRA Fine-Tuning for Singlish-to-Sinhala Transliteration and Code-Mix Translation},
  author={Gunarathna, Savinu},
  year={2025},
  url={https://huggingface.co/savinugunarathna/Gemma3-Singlish-Sinhala-CodeMix}
}

Related Work

@article{sumanathilaka2025swa,
  title={Swa-bhasha Resource Hub: Romanized Sinhala to Sinhala Transliteration Systems and Data Resources},
  author={Sumanathilaka, Deshan and Perera, Sameera and Dharmasiri, Sachithya and Athukorala, Maneesha and Herath, Anuja Dilrukshi and Dias, Rukshan and Gamage, Pasindu and Weerasinghe, Ruvan and Priyadarshana, YHPP},
  journal={arXiv preprint arXiv:2507.09245},
  year={2025}
}

@article{ranasinghe2022sold,
  title={SOLD: Sinhala Offensive Language Dataset},
  author={Ranasinghe, Tharindu and Anuradha, Isuri and Premasiri, Damith and Silva, Kanishka and Hettiarachchi, Hansi and Uyangodage, Lasitha and Zampieri, Marcos},
  journal={arXiv preprint arXiv:2212.00851},
  year={2022}
}

@inproceedings{Nsina2024,
  author={Hettiarachchi, Hansi and Premasiri, Damith and Uyangodage, Lasitha and Ranasinghe, Tharindu},
  title={{NSINA: A News Corpus for Sinhala}},
  booktitle={The 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year={2024},
  month={May},
}

Acknowledgments

• Google for the Gemma model family
• Swa-bhasha Resource Hub for the phonetic transliteration dataset
• Hugging Face for the transformers ecosystem
• Training compute provided by Kaggle (P100 GPU)