tinystories-gpt-from-scratch

TinyStories GPT from Scratch

A small GPT-style language model trained from scratch on TinyStories. It is an educational model for generating short, simple English stories.

Model details

• Architecture: decoder-only transformer
• Parameters: 12,297,984
• Layers: 6
• Attention heads: 6
• Embedding size: 384
• Context length: 256 tokens
• Tokenizer: byte-level BPE, 4,096 tokens
• Framework: PyTorch

The exported checkpoint was saved at training step 1,000. Its recorded training loss was 3.11046 and recorded validation loss was 3.07700. These values are checkpoint metadata, not a broader benchmark.

Usage

import sys
from pathlib import Path

import torch
from huggingface_hub import snapshot_download
from tokenizers import Tokenizer

model_dir = Path(snapshot_download("Haider92/tinystories-gpt-from-scratch"))
sys.path.insert(0, str(model_dir))

from inference_utils import generate_story, load_exported_model

device = torch.device("cpu")
model = load_exported_model(
    model_dir / "config.json",
    model_dir / "model.pt",
    device,
)
tokenizer = Tokenizer.from_file(
    str(model_dir / "tinystories_tokenizer.json")
)

result = generate_story(
    model,
    tokenizer,
    prompt="Once upon a time",
    target_tokens=120,
    extra_tokens=80,
    temperature=0.8,
    top_k=40,
    device=device,
)
print(result.story)

Intended use

This model is intended for education, experimentation, and short TinyStories-style text generation. It is not a general-purpose assistant.

Limitations

The model is small and trained on a constrained synthetic-story dataset. Outputs may be repetitive, inconsistent, incomplete, or factually incorrect. It has no application-specific safety guarantees.

Dataset and license

TinyStories is distributed under CDLA-Sharing-1.0. No dataset files are included in this repository.

The exported model weights and included source code are available under the MIT License.

Links

• Source code
• Interactive demo