tinystories-pos-embed

TinyStories toy transformer: learned vs no positional embeddings

Two variants, identical except for the positional embedding:

• learned
• none

Model

Training

AdamW, lr 1e-3, weight decay 1e-2, batch size 32, 500 steps per "epoch" (each epoch is ~2M tokens, i.e. fresh data — no repeats). Test loss is mean next-token cross-entropy on a held-out 1000-sequence split.

Checkpoint format

Each .pt is a plain dict, loadable with weights_only=True:

{
    "cfg": {...},         # model config incl. "pos_embed": "learned" | "none"
    "state_dict": {...},  # CPU tensors
    "args": {...},        # training args (epochs, lr, seed, ...)
    "test_loss": float,   # final held-out loss
}