scratch-gpt-mcq

Scratch GPT — MCQ Top-3 Ranker

A decoder-only transformer implemented from scratch in PyTorch — no pretrained weights, no transformers model class. Pretrained as a character/BPE language model on a Wikipedia corpus, then fine-tuned with a linear scoring head to rank five multiple-choice options (A–E) and emit the top three, scored with MAP@3.

Interactive demo: https://huggingface.co/spaces/your-hf-username/scratch-gpt-mcq

Architecture

Built up from the primitives: single attention head → multi-head attention → position-wise MLP → pre-norm residual block → full model.

Training

Stage 1 — language-model pretraining. Next-token cross-entropy on a Wikipedia corpus, AdamW at 3e-4, batch 64, 4000 iterations, 10% held out for validation. Reported as bits-per-character so the BPE and character tokenizers are comparable on the same axis.

Stage 2 — multiple-choice fine-tuning. Each (question, option) pair is encoded as question + "\n" + option, tail-truncated to the context window so the option always survives. The hidden state at the last real token goes through a linear head to one scalar; the five scalars are softmaxed and trained with cross-entropy against the correct letter. AdamW at 3e-4, 3 epochs, 8 questions (40 sequences) per step. The train/validation split is grouped by prompt so no prompt appears on both sides.

Files

Usage

import torch
from huggingface_hub import hf_hub_download
import modeling   # download modeling.py from this repo

REPO = "your-hf-username/scratch-gpt-mcq"
tok = modeling.load_tokenizer(hf_hub_download(REPO, "tokenizer_bpe.json"))
mc, cfg = modeling.load_mc_model(hf_hub_download(REPO, "gpt_wiki_mc.pt"))

top3, scores = modeling.rank_options(
    mc, tok, cfg,
    "Which phenomenon explains the bending of light around a massive object?",
    ["Gravitational lensing", "Rayleigh scattering", "Total internal reflection",
     "The photoelectric effect", "Bremsstrahlung"],
)
print(" ".join(top3))     # e.g. "A C B"  -> submission format

Limitations

5.24M parameters trained on a small corpus for a few thousand steps. It is an architectural exercise, not a competitive QA system: the language-model samples are Wikipedia-flavoured but not factual, and the multiple-choice head generalises only to questions resembling the competition distribution. Not for any use where being wrong matters.