Tern

Tern

A GPT-2 small architecture language model, trained entirely from random initialization on a single consumer GPU (RTX 4070 SUPER, 12GB VRAM) — no pretrained weights anywhere in this checkpoint's lineage. This is the fine-tuned checkpoint (base pretraining + SFT), the one meant to actually be talked to.

Uploading this is meant to let people skip the training run (base pretraining alone takes multiple days on a single consumer GPU) and start from a finished, working checkpoint.

Architecture

Standard GPT-2 small config, unmodified: 12 transformer layers, 12 attention heads, 768 embedding dimensions, 1024 token context window, 123.6M parameters, GPT-2's own BPE vocabulary (tiktoken, 50,257 tokens). The implementation is nanoGPT's own model.py, unmodified — this project's own "from scratch" is about the weights (random init, not fine-tuned from anyone else's), not about reinventing the transformer.

Training

Stage 1 — base pretraining: OpenWebText, ~9 billion tokens, one pass, 25,000 iterations, batch size 6 with gradient accumulation 64 (393,216 tokens per optimizer step), bfloat16 mixed precision, AdamW with cosine decay and linear warmup. Final validation loss: 3.07. OpenAI's own original GPT-2 checkpoint scores 3.11 on this same dataset — this model, trained from scratch on one consumer GPU, matches that baseline.

Stage 2 — supervised fine-tuning: 6,000 iterations continuing from the base checkpoint, teaching three fixed prompt/answer template shapes rather than new facts (the "superficial alignment hypothesis," LIMA, 2023):

Context: <retrieved passage>
Question: <question>
Answer: <paraphrase of the passage, never copied verbatim>

Context: (none)
Question: <question that needs a tool>
Answer: CALL: calculator(<expression>)

Context: (none)
Question: <question with nothing to answer from>
Answer: I don't have information about that.

Final SFT loss: train 0.29, val 0.52. The RAG shape is trained exclusively on real, independently-written Simple English Wikipedia paraphrases, never the retrieved passage copied verbatim — no generative AI produced any training data anywhere in this pipeline.

What this checkpoint is (and isn't) good for

This checkpoint alone gives you a small, real, from-scratch language model that can complete text and, if you feed it prompts in the exact three shapes above, follow that template. It was not trained to be a general-purpose open-domain knowledge model — the no-match shape trains it to say "I don't have information about that." for a real factual question it has no Context: block for, on purpose, rather than to guess. Its intended use is paired with a retrieval layer that supplies real Context: text (this is what the full project, not just this checkpoint, does): a self-built FAISS + cross-encoder retrieval pipeline over a 6.4 million article Wikipedia corpus, a small calculator/datetime tool registry, and a chat loop that never lets the model's answer be a verbatim copy of anything it was shown.

Loading it

This is a raw nanoGPT-format checkpoint (state_dict + model_args + iter_num + best_val_loss), not a transformers-compatible model — it needs nanoGPT's own GPT/GPTConfig classes to load, not AutoModelForCausalLM:

import torch
from model import GPTConfig, GPT  # nanoGPT's model.py

checkpoint = torch.load("ckpt.pt", map_location="cpu", weights_only=False)
gptconf = GPTConfig(**checkpoint["model_args"])
model = GPT(gptconf)

state_dict = checkpoint["model"]
unwanted_prefix = "_orig_mod."
for k, v in list(state_dict.items()):
    if k.startswith(unwanted_prefix):
        state_dict[k[len(unwanted_prefix):]] = state_dict.pop(k)
model.load_state_dict(state_dict)
model.eval()

Tokenize with tiktoken's gpt2 encoding (standard GPT-2 BPE, nothing custom).

Honest limitations

124M parameters is small. Without a real Context: block, it does not have reliable general world knowledge, and for an ordinary factual question in that shape it is trained to say exactly that ("I don't have information about that.") rather than hallucinate a confident-sounding wrong answer — a deliberate design choice, not an oversight.

That is trained behavior on the template shapes above, not a guarantee against every possible input. Two real limits worth knowing:

• Feed it something far outside what it was trained on (a wall of repeated characters, thousands of tokens of noise) and it will still produce a fluent, empty sentence instead of the refusal.
• The whole prompt has to fit the 1024 token context window. If it does not, nanoGPT's generate() crops from the front, which silently removes the Context: ... / Question: ... template and leaves the model completing raw text with nothing to follow. Truncate long questions yourself before generating (the project's own chat loop does this explicitly) rather than letting the window do it for you.

It has no memory across turns and was never trained on multi-turn conversation, instructions outside its three template shapes, code, or opinions.

License

MIT.