mgpt2-sft

mgpt2-sft — Multilingual GPT-2 (Instruction-Tuned)

mgpt2 fine-tuned on 30,000 multilingual instruction–response pairs across 5 language variants: English, Hindi (Devanagari), Hindi (Latin transliteration), Kannada (Kannada script), and Kannada (Latin transliteration). Training data from ai4bharat/indic-align (Anudesh, Dolly-T, OpenAssistant-T).

Built on top of the pretrained mgpt2 base — same 124M architecture, same custom multilingual tokenizer. Uses masked cross-entropy (loss computed over response tokens only).

Quick start

import sys, torch
import torch.nn.functional as F
from huggingface_hub import snapshot_download

local = snapshot_download("ace-1/mgpt2-sft")
sys.path.insert(0, local)
from model import GPT
from tokenizer.regex_tokenizer import RegexTokenizer

ckpt = torch.load(f"{local}/pytorch_model.pt", weights_only=False, map_location="cpu")
model = GPT(ckpt["config"])
model.load_state_dict(ckpt["model"])
model.eval()

enc = RegexTokenizer()
enc.load(f"{local}/tokenizer/artifacts/mgpt2.model")

# Prompt: plain text, no special template needed
prompts = [
    "What is the capital of Karnataka?",                   # English
    "कर्नाटक की राजधानी क्या है?",                          # Hindi (Devanagari)
    "ಕರ್ನಾಟಕದ ರಾಜಧಾನಿ ಯಾವುದು?",                           # Kannada script
]

for prompt in prompts:
    ids = enc.encode(prompt)
    x = torch.tensor(ids, dtype=torch.long).unsqueeze(0)
    with torch.no_grad():
        for _ in range(120):
            logits, _ = model(x[:, -1024:])
            probs = F.softmax(logits[:, -1, :] / 0.7, dim=-1)
            next_id = torch.multinomial(probs, num_samples=1)
            if next_id.item() == 50256: break
            x = torch.cat([x, next_id], dim=1)
    print(f"Prompt : {prompt}")
    print(f"Response: {enc.decode(x[0, len(ids):].tolist())}")
    print()

Intended use

Good for:

• Multilingual Q&A and instruction following (en/hi/kn, native + romanised scripts)
• Downstream fine-tuning starting point for Indic NLP tasks
• Research: multilingual instruction tuning at small scale

Not for: Safety-critical applications. Native-script variants (Devanagari, Kannada) are more reliable than transliterated Latin variants, which are prone to mid-generation script drift (known limitation — see training notes).

Model details

Training configuration

Evaluation

SFT val PPL is measured on the SFT held-out set (narrower domain) and is not comparable to the pretrain LM eval PPL (12.4), which measures general language modelling ability.

Training data

30,000 examples total. 90/10 train/val split. Masked CE — loss computed over response tokens only.

Tokenizer

Custom multilingual regex + BPE tokenizer (mgpt2), trained on the same corpus mixture. Same vocabulary size as tiktoken-gpt2 (50,257 tokens), but with Indic-aware merge priorities:

Tokenizer published separately: ace-1/mgpt2-tokenizer

Known limitations

• Transliterated Latin script drift. hin_Latn and kan_Latn may switch scripts mid-generation. Cause: ASCII tokens shared with English; no Unicode anchor. Mitigated but not eliminated at this data scale.
• 124M parameters. Factual accuracy and multi-step reasoning are limited.
• No safety alignment. The SFT model was trained on benign instruction data only; it may attempt to answer harmful prompts. Use the DPO variant for light safety alignment.
• Research checkpoint — not evaluated for production use.

Citation

@misc{mgpt2,
  title     = {mgpt2: Multilingual GPT-2 with custom Indic tokenizer},
  year      = {2026},
  note      = {Pretrain → SFT → DPO pipeline for English/Hindi/Kannada},
  url       = {https://huggingface.co/ace-1/mgpt2-sft}
}