experimental_ztr-l-1

ZTR-L-1

ZTR-L-1 is the first language model I have ever made from scratch. It is a 303-million parameter transformer that I pretrained on one billion tokens and then fine-tuned so it can hold a simple conversation.

I wrote the whole stack myself in TypeScript (the tokenizer), the training code, and the inference engine, and it all runs on WebGPU. Every bit of the training happened on a single RTX 5070 Ti lol.

It is extremely unreliable but slightly conversable for a few messages. Treat it as a research toy, not an assistant.

Talking to it

The weights use a custom checkpoint format and run with the ZTR CLI (Node.js + WebGPU):

node bin/ztr.mjs chat --model <this-folder> --backend gpu \
  --greedy --repetition-penalty 1.2 --repeat-ngram 3 --max-new-tokens 256

The tokenizer ships inside this folder and is picked up automatically. A conversation looks like this:

You: What is the capital of France?
ZTR-L-1: The capital of France is Paris.

Files

• model.bin — the weights
• manifest.json — describes the architecture and how to load the weights
• ztr-l-1-32k.json — the 32k-vocab BPE tokenizer I trained with the model

Part of the Zeterelle series — models I am building one at a time, on one GPU.