Tiny-GPT

TinyGPT

A small GPT-style decoder-only language model, trained from scratch on TinyStories.

• Architecture: 8-layer, 16-head, 1024-dim decoder-only transformer (fused QKV attention via scaled_dot_product_attention, GELU MLP, pre-LayerNorm, tied input/output embeddings).
• Tokenizer: character-level over ASCII code points 0–127 (vocab_size=128), i.e. id == ord(char).
• Context length: 1024 tokens.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "yahya94812/Tiny-GPT"

tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)

inputs = tokenizer("Once upon a time", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=200, do_sample=True, temperature=0.8)
print(tokenizer.decode(out[0]))

trust_remote_code=True is required because this model uses a custom architecture (modeling_tinygpt.py) rather than one built into transformers.

Limitations

This is a small research/educational model trained on TinyStories; it generates simple, short children's-story-style text and will not perform general-purpose language tasks.