luffy-gpt

Luffy GPT

49.4M parameter GPT transformer trained from scratch on One Piece (Luffy) dialogue, fine-tuned with 5,004 SFT conversation pairs using style token conditioning and GPT-2 subword tokenizer.

Models

Architecture

• Decoder-only GPT built from scratch in PyTorch
• n_embd=384, n_head=6, n_layer=6, context_size=256
• KV cache, top-k/top-p sampling, temperature, repetition penalty
• Style embedding for switching between corpus and conversation mode
• EOS token for controlled generation stopping
• Cosine LR decay with warmup, weight decay, early stopping

Try it out

This is not a perfect chatbot. It's a small model trained from scratch on limited data. Works best with direct questions.

Prompts that work well:

• Who are you?
• What is your dream?
• Are you hungry?
• Are you scared?
• Who is Zoro?
• Who is Shanks?
• What happened to Ace?
• Are you a hero?

Run locally

# chat with best model (GPT-2 tokenizer)
python gpt.py --input dataset/processed/corpus_clean.txt \
  --eval luffy_gpt_gpt2tok_finetuned.pth --interactive --style 1 \
  --temperature 0.8 --top-k 50 --top-p 0.9 --repetition-penalty 1.2 \
  --gpt2-tokenizer

# pretrain from scratch
python gpt.py --input dataset/processed/corpus_clean.txt \
  --train luffy_gpt_gpt2tok.pth --epoch 25000 --gpt2-tokenizer \
  --warmup-steps 500 --weight-decay 0.1

# fine-tune
python gpt.py --input dataset/processed/corpus_clean.txt \
  --finetune luffy_gpt_gpt2tok_finetuned.pth \
  --sft-input dataset/luffy_sft.txt --pretrained luffy_gpt_gpt2tok.pth \
  --epoch 10000 --lr 5e-5 --warmup-steps 200 --weight-decay 0.1 \
  --gpt2-tokenizer

Sample outputs

You: who are you?
Luffy: I'm Monkey D. Luffy!

You: are you scared?
Luffy: Nope.

You: are you okay?
Luffy: Not really.

Tokenizer comparison

Built as part of the Build Fellowship program.