tokilm-9m-chat

<p align="center"> <img src="https://github.com/skorotkiewicz/tokilm/blob/main/assets/tokilm.png?raw=true" alt="TokiLM" width="300"/> </p> <p align="center"> <a href="https://github.com/skorotkiewicz/tokilm"><img src="https://img.shields.io/badge/GitHub-tokilm-181717?logo=github" alt="GitHub"/></a>&nbsp; <a href="https://huggingface.co/datasets/finnnnnnnnnnnn/toki-pona-sentences"><img src="https://img.shields.io/badge/🤗_Dataset-toki--pona--sentences-blue" alt="Dataset"/></a>&nbsp; <a href="https://colab.research.google.com/github/skorotkiewicz/tokilm/blob/main/train_tokilm.ipynb"><img src="https://img.shields.io/badge/Open_in-Colab-F9AB00?logo=googlecolab" alt="Colab"/></a> </p>

TokiLM — 9M Parameter Toki Pona Language Model

A ~9M parameter LLM trained from scratch that speaks Toki Pona.

This project exists to show that training your own language model is not magic. One small repo, a few minutes on a T4 GPU, and you have a working LLM built from scratch.

Example

You> toki
TokiLM> toki a. sina pona. mi wile toki pona.

You> moku li pona
TokiLM> moku li pona tawa mi. mi moku e kili.

You> sina pilin seme
TokiLM> mi pilin pona. mi lukin e suno.

Architecture

No GQA, no RoPE, no SwiGLU, no early exit. As simple as it gets.

Training

• Data: finnnnnnnnnnnn/toki-pona-sentences — a corpus of Toki Pona sentences
• Steps: 10,000
• Optimizer: AdamW (cosine LR schedule)
• Hardware: T4 GPU (~5 min)
• No system prompt — the language is baked into the weights

Usage

python -m tokilm chat --checkpoint pytorch_model.bin --tokenizer tokenizer.json

Or programmatically:

from tokilm.inference import TokiInference

engine = TokiInference("pytorch_model.bin", "tokenizer.json")
r = engine.chat_completion([{"role": "user", "content": "toki"}])
print(r["choices"][0]["message"]["content"])

Links

• Repo: github.com/skorotkiewicz/tokilm
• Dataset: huggingface.co/datasets/finnnnnnnnnnnn/toki-pona-sentences

License

MIT