lilstory

lilstory

a tiny 8m parameter llama-style model trained from scratch on tinystories. it writes short simple stories that mostly make sense and then kinda wander off.

run it

ollama run navthings/lilstory

or use lilstory.gguf with llama.cpp. the transformers version (model.safetensors + config + tokenizer) is in here too.

or try it in your browser: https://navthings.github.io/playground/

details

4 layers, 256 wide, 4 attention heads, 8000 token bpe tokenizer i trained myself. trained on a macbook with adamw, lr 1e-3, batch 32.

used in my paper: the effect of corpus size on the performance of a llm

code: https://github.com/navthings/lilstory