eeny-tinystories-999k

eeny — a coherent TinyStories generator in 999,328 parameters

eeny is a tiny language model that writes short children's stories — and runs in a browser tab, offline, with no GPU. At 999,328 total parameters it is, to our knowledge, among the smallest coherent story models, and it beats TinyStories-1M (a model ~3× larger by non-embedding params) on a fair, tokenizer-independent metric.

Numbers (all measured)

Bits-per-byte is tokenizer-independent (normalize cross-entropy by UTF-8 bytes), so the comparison is fair across different tokenizers.

What it is

• Architecture: small decoder-only transformer (dim 88, 7 layers, GQA, RoPE, SwiGLU, tied embeddings), 4096-token BPE tokenizer trained on TinyStories.
• Trained by knowledge distillation from a small full-precision teacher on the TinyStories dataset, then exported to int8 (per-row) for the Sprapp in-browser WASM engine. int8 is near-lossless for this fp model (ternary post-training-quantization destroyed it — int8 preserves it).

Sample (greedy, in-engine)

Once upon a time, there was a little  →  girl named Lily. She loved to play outside in the sun and
  watch the birds fly high up in the sky. One day, she saw an old man sitting ...
The cat sat on the                    →  tree and said, "I'm sorry I was so sad. Can you forgive me?"
  The cat smiled and gave the cat a big hug. From that day on, the cat always shared ...

Files

Limitations

• Domain: TinyStories only (simple children's stories). Not a general LM.
• Tiny: it confabulates and has no world knowledge beyond the TinyStories distribution.

Part of the Sprapp project — offline on-device tiny LMs in the browser (eeny / meeny / miny family). Trained on TinyStories (Eldan & Li, 2023).