mgpt2-dpo

mgpt2-dpo — Multilingual GPT-2 (Preference-Aligned)

Recommended model from this project. mgpt2-sft further aligned with Direct Preference Optimization (DPO, β=0.1) on 13,500 toxic preference pairs from ai4bharat/indic-align (HHRLHF-T). Chosen responses are Llama2-70B-Chat safety refusals; rejected responses are raw pretrained-model continuations.

DPO increased the log-probability of safety-refusal responses by +6.1% and decreased the log-probability of rejected responses by −16.8% relative to the SFT checkpoint. Generation comparisons show the SFT model attempts to comply with harmful prompts; the DPO model redirects. See the project report for full analysis.

Quick start

import sys, torch
import torch.nn.functional as F
from huggingface_hub import snapshot_download

local = snapshot_download("ace-1/mgpt2-dpo")
sys.path.insert(0, local)
from model import GPT
from tokenizer.regex_tokenizer import RegexTokenizer

ckpt = torch.load(f"{local}/pytorch_model.pt", weights_only=False, map_location="cpu")
model = GPT(ckpt["config"])
model.load_state_dict(ckpt["model"])
model.eval()

enc = RegexTokenizer()
enc.load(f"{local}/tokenizer/artifacts/mgpt2.model")

prompts = [
    "Explain what photosynthesis is.",             # English
    "प्रकाश संश्लेषण क्या है?",                    # Hindi (Devanagari)
    "ದ್ಯುತಿಸಂಶ್ಲೇಷಣೆ ಎಂದರೇನು?",                    # Kannada script
]

for prompt in prompts:
    ids = enc.encode(prompt)
    x = torch.tensor(ids, dtype=torch.long).unsqueeze(0)
    with torch.no_grad():
        for _ in range(120):
            logits, _ = model(x[:, -1024:])
            probs = F.softmax(logits[:, -1, :] / 0.7, dim=-1)
            next_id = torch.multinomial(probs, num_samples=1)
            if next_id.item() == 50256: break
            x = torch.cat([x, next_id], dim=1)
    print(f"Prompt : {prompt}")
    print(f"Response: {enc.decode(x[0, len(ids):].tolist())}")
    print()

Intended use

Good for:

• Multilingual instruction following with light safety alignment (en/hi/kn)
• Research: DPO alignment dynamics at 124M scale
• Demo of end-to-end LLM pipeline: pretrain → SFT → DPO

Not for: Production safety-critical applications. Alignment is format-preference alignment (coherent refusals vs incoherent noise), not full safety alignment. At 124M parameters the pretrained model could not generate coherent harmful content, so the DPO preference signal is weaker than production RLHF setups.

Model details

Training configuration

Evaluation

100% win-rate reflects format-preference alignment (coherent refusals vs word-salad), not full safety alignment. See project report for full generation comparison.

Training data

13,500 train / 1,499 val pairs. Source: ai4bharat/indic-align HHRLHF-T config.

Tokenizer

Custom multilingual regex + BPE tokenizer (mgpt2), trained on the same corpus mixture. Same vocabulary size as tiktoken-gpt2 (50,257 tokens), but with Indic-aware merge priorities:

Tokenizer published separately: ace-1/mgpt2-tokenizer

Known limitations

• Format-preference alignment, not full safety alignment. At 124M parameters, the pretrained model generates incoherent text for toxic prompts, so the DPO preference signal trains format preference (coherent refusals vs noise) rather than genuine safety reasoning.
• Transliterated Latin script drift (inherited from SFT checkpoint) — hin_Latn/kan_Latn may switch scripts mid-generation.
• 124M parameters. Factual accuracy and multi-step reasoning are limited.
• Research checkpoint — not evaluated for production use.

Citation

@misc{mgpt2,
  title     = {mgpt2: Multilingual GPT-2 with custom Indic tokenizer},
  year      = {2026},
  note      = {Pretrain → SFT → DPO pipeline for English/Hindi/Kannada},
  url       = {https://huggingface.co/ace-1/mgpt2-dpo}
}