v11-tinystories-115m-base

v11-tinystories-115m-base

A 115.1M-parameter decoder-only transformer trained from scratch on TinyStories for 16M tokens, using the v11 tokenizer.

This is the base pretrain only: phase 1, no frozen-FFN attention retrain, and no maths mid-training. It writes competent children's-story English and it cannot do arithmetic — it narrates straight past the place a number belongs rather than putting a wrong number there. That is the intended state of this checkpoint, not a defect: it is the starting point for the mid-training experiments in tinystories-train-video.

Loading

Not an AutoModel — TinyModel is a 3-file Gemma-shaped decoder (RMSNorm, RoPE, GQA, gated FFN, tied embeddings), shipped in this repo under tiny_model_v11/.

import json, torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from transformers import AutoTokenizer

path = snapshot_download("chrishayuk/v11-tinystories-115m-base")

import sys; sys.path.insert(0, path)
from tiny_model_v11 import TinyModel

cfg = json.load(open(f"{path}/config.json"))
model = TinyModel(
    vocab_size=cfg["vocab_size"], dim=cfg["dim"], n_layers=cfg["n_layers"],
    ffn_dim=cfg["ffn_dim"], n_heads=cfg["n_heads"], n_kv_heads=cfg["n_kv_heads"],
    max_seq=cfg["max_seq"],
)
model.load_state_dict(load_file(f"{path}/model.safetensors"))
model.eval()

tok = AutoTokenizer.from_pretrained("chrishayuk/v11-tinystories-115m-base")
ids = [tok.convert_tokens_to_ids("<s>")] + tok("Once upon a time", add_special_tokens=False)["input_ids"]
for _ in range(40):
    logits = model(torch.tensor([ids[-cfg["max_seq"]:]]))[0, -1]
    ids.append(int(logits.argmax()))
print(tok.decode(ids))

That exact round-trip — download, build, load, generate — is run against the uploaded files at publish time. It is not an untested snippet.

Architecture

The embedding table is 36.5M parameters — 32% of the whole model is its vocabulary lookup, which is what a 71k vocabulary costs at this width.

rope_freqs is a complex64 buffer in the state dict. It is derived from dim/n_heads/max_seq and recomputed on construction, so it round-trips but carries no learned information.

Identity

Identity is the content hash, not the Hub revision — re-pushing identical bytes mints a new commit oid, and a README edit does too. Join on these:

The tokenizer sha is the same value this checkpoint's training run wrote into its meta.json as tokenizer_hash, and the same one chrishayuk/v11-tokenizer publishes. A checkpoint driven by a different tokenizer produces fluent nonsense rather than an error, so that join is checked mechanically at publish time, not asserted here.

Training

The dataset revision is pinned, so the document set is reproducible and held-out text can be shown never to have been trained on.

Capability emergence

emergence.json carries the generations captured at each milestone, plus the loss trace. Greedy, 30 new tokens, same prompts throughout:

Only the final row's weights are published here; the earlier rows are the same run mid-flight.

What this model cannot do

It cannot do arithmetic. Nothing in TinyStories teaches addition, and number words in that corpus are narrative texture rather than quantities — "once upon a time there were two" is an idiom the model learns the way it learns "happily ever after".

The interesting part is how it fails. It does not answer with a wrong number; it carries on telling the story, straight past the place a number belongs.

Its own generation at the final milestone, greedy, from this run:

Lily had three apples. Tom gave her four more. Now Lily has a big piece of paper. She likes the color. Tom likes the color. He likes the

It has also seen 16M tokens, which is not converged. Continued training on more or less anything improves it, so do not read an improvement after mid-training as evidence that the mid-training data helped specifically.

Intended use

Research and teaching on compact language models: mid-training, tool-use / tool-call training, and measuring the difference between memorising a distribution and learning an algorithm. Not intended for any production use, and it has no safety training of any kind.

Provenance

provenance.json records the model and tokenizer hashes, the corpus pin, and the run config. The chuk-datasets catalog holds a content-addressed, bit-reproducible tokenization of the same corpus (tiny-model/v11-rust-tokenized-phase1); these weights came from the pinned HF revision instead, which is reproducible given the seed but not bit-identical. That distinction is recorded rather than glossed.