tinystories-game-boy-color

TinyStories 260K for Game Boy Color

This repository packages TinyStories-260K artifacts and a custom Game Boy Color runtime for local language-model inference on stock GBC hardware.

The original project is a proof-of-concept GBDK-2020 ROM that tokenizes a prompt on-device, runs integer/fixed-point transformer inference, and generates a short continuation using a 512-token TinyStories tokenizer. It is intentionally tiny, slow, and imperfect.

Contents

• model/stories260K.bin: TinyStories-260K checkpoint in the llama2.c format.
• model/tok512.bin: 512-token TinyStories tokenizer.
• source/src/: desktop reference runners, tokenizer code, and GBC runtime.
• source/tools/: asset fetch, quantization, chunk generation, and simulation scripts.
• source/FLASHCART_README.txt: flashcart notes from the source project.
• source/NOTICE.md: upstream artifact notices.

The source project can build build/TINYSTORY.GB for an MBC5-compatible flashcart, but this prepared package does not include a prebuilt ROM.

Model Lineage

The checkpoint and tokenizer come from Andrej Karpathy's karpathy/tinyllamas TinyStories 260K artifacts. The runtime adapts the tiny Llama/TinyStories model shape to an 8-bit handheld target.

Model shape:

• dim=64
• hidden_dim=172
• layers=5
• heads=8
• kv_heads=4
• vocab=512

Runtime

The GBC runtime includes:

• MBC5 bank-switched cartridge data for the model
• on-device BPE tokenization
• integer/fixed-point transformer inference
• RoPE, attention, RMSNorm, and SwiGLU-style MLP code
• cartridge SRAM KV cache
• on-screen keyboard prompt entry
• greedy argmax decoding

Build

The full build flow lives in the GitHub project:

git clone https://github.com/maddiedreese/gbc-transformer.git
cd gbc-transformer
make fetch-assets
make gbc-rom

By default the Makefile expects GBDK-2020 at tools/vendor/gbdk-4.5.0, or you can pass GBDK_HOME=/path/to/gbdk.

Limitations

• Very slow on real hardware.
• Output quality is rough because of heavy quantization and fixed-point approximations.
• Context is capped at 16 tokens.
• Greedy decoding only.
• Built for the TinyStories-260K model shape above.

Citation and Links

• GitHub: https://github.com/maddiedreese/gbc-transformer
• Base artifacts: https://huggingface.co/karpathy/tinyllamas/tree/main/stories260K
• TinyStories paper: https://arxiv.org/abs/2305.07759
• llama2.c: https://github.com/karpathy/llama2.c
• GBDK-2020: https://github.com/gbdk-2020/gbdk-2020