gpt-10m-dpo-sharegpt

GPT-10M-DPO-ShareGPT

A tiny 10-million parameter GPT-style model trained from scratch, fine-tuned using Supervised Fine-Tuning (SFT) on ShareGPT conversations, and aligned via Direct Preference Optimization (DPO) using preferences judged by Gemini 3.1 Flash Lite.

Model Details

• Architecture: Decoder-only Transformer (GPT)
• Parameters: ~10 Million
• Layers: 6
• Heads: 6
• Embedding Dimension: 384
• Context Length: 512 tokens
• Vocabulary Size: 50,257 (custom BPE tokenizer)

Training Stages

1. Pretraining: Continued pretraining for ~24.6M tokens (part of a full epoch over a 95M curated flat-packed token dataset consisting of 50% code, 20% knowledge, and 10% textbooks).
2. Supervised Fine-Tuning (SFT): Fine-tuned for 500 steps using float16 precision on 10,000 multi-turn conversations from the ShareGPT dataset. Best SFT validation loss reached 3.4184.
3. Direct Preference Optimization (DPO): Aligned on 255 preference pairs. Prompts were sampled from SFT logs, completions were generated by the SFT model, and preferred responses were judged by Gemini 3.1 Flash Lite.

How to Load and Sample

You can use the custom PyTorch definition in model.py and tokenizer.json to load the model:

import torch
from tokenizers import Tokenizer
from model import GPT

# Load model checkpoint
checkpoint = torch.load("dpo_model.pt", map_location="cpu")
config = checkpoint["config"]
model = GPT(config)
model.load_state_dict(checkpoint["model"])
model.eval()

# Load tokenizer
tokenizer = Tokenizer.from_file("tokenizer.json")

# Generate text
prompt = "<|im_start|>user\nWrite a python function to check if a number is prime.<|im_end|>\n<|im_start|>assistant\n"
x = torch.tensor(tokenizer.encode(prompt).ids, dtype=torch.long).unsqueeze(0)
y = model.generate(x, max_new_tokens=150, temperature=0.8, top_k=50)
print(tokenizer.decode(y[0].tolist()))