teensy-0

Teensy-0

Teensy-0 is a skinny-deep decoder-only language model. It was trained from scratch on the OpenWebText corpus using a custom training pipeline.

The architecture and training pipeline are modified from NanoGPT by Andrej Karpathy, adapted specifically for the Teensy model family.

Model Details

Teensy-0 follows the naming convention 30L_8H_64BLK_12.35M:

• 30L — 30 transformer layers
• 8H — 8 attention heads
• 64BLK — 64-token context length (block size)
• 12.35M — ~12.35 million parameters

Training

Teensy-0 was trained from scratch on the OpenWebText corpus:

• ~9B tokens
• 20,000 iterations
• batch size 12, gradient accumulation 40
• AdamW, peak learning rate 6e-4 with warmup and decay
• Best validation loss: 4.5432

The training run was produced with the Teensy training code, not by fine-tuning an existing model. The original data, notebooks, and training scripts are archived in the sibling teensy-1/ directory.

Files in this Repository

Installation

pip install -r requirements.txt

Requirements:

• torch>=2.0
• tiktoken>=0.5
• numpy>=1.24
• safetensors>=0.4

Device Notes

CPU is strongly recommended for inference. This checkpoint was trained with the PyTorch MPS backend, and inference on MPS can produce degraded output: garbled text, stray <|endoftext|> tokens, and invalid UTF-8 byte sequences. All inference scripts therefore default to cpu. Pass --device=mps explicitly if you want to use Apple Silicon GPU, but expect lower-quality output.

Usage

1. Interactive streaming REPL (.pt checkpoint)

python repl.py --device=cpu --dtype=float32

Type a prompt and Teensy will stream tokens back token-by-token. Generation stops automatically when the model emits the GPT-2 <|endoftext|> token. Type exit or press Ctrl+C to quit.

2. Sample from a prompt (.pt checkpoint)

python sample.py --device=cpu --dtype=float32 \
  --start="Once upon a time" \
  --max_new_tokens=200 \
  --num_samples=3

3. Generate from the safetensors export

python sample_hf.py --device=cpu --dtype=float32 \
  --prompt="Once upon a time" \
  --max_new_tokens=200

4. Interactive streaming REPL for the safetensors export

python repl_hf.py --device=cpu --dtype=float32

Streams tokens and stops automatically at <|endoftext|>.

4. Load in your own code

From the .pt checkpoint

import torch
from model import TeensyConfig, TeensyLM, adapt_nanogpt_weights

checkpoint = torch.load("checkpoints/teensy-0.pt", map_location="cpu")
cfg = TeensyConfig(**checkpoint["model_args"])
model = TeensyLM(cfg)
state_dict = adapt_nanogpt_weights(checkpoint["model"])
model.load_state_dict(state_dict)
model.eval()

From the safetensors export

import json
import torch
from safetensors.torch import load_file
from model import TeensyConfig, TeensyLM

with open("exported/config.json") as f:
    cfg = TeensyConfig(**json.load(f))

model = TeensyLM(cfg)
model.load_state_dict(load_file("exported/model.safetensors"))
model.eval()

5. Re-export to safetensors

python export_hf.py --out_dir=checkpoints --export_dir=exported

This writes exported/model.safetensors (~75 MB, ~49% smaller than the .pt checkpoint) and exported/config.json.

Downloading without Cloning the Whole Repo

If you only want the model files, you can use the huggingface_hub library:

pip install huggingface_hub

Download the .pt checkpoint

from huggingface_hub import hf_hub_download

path = hf_hub_download(
    repo_id="Teensy/teensy-0",
    filename="checkpoints/teensy-0.pt",
    local_dir="./teensy-0"
)
print(path)

Download the safetensors export

from huggingface_hub import hf_hub_download

config_path = hf_hub_download(
    repo_id="Teensy/teensy-0",
    filename="exported/config.json",
    local_dir="./teensy-0"
)
weights_path = hf_hub_download(
    repo_id="Teensy/teensy-0",
    filename="exported/model.safetensors",
    local_dir="./teensy-0"
)
print(config_path, weights_path)

Download everything

huggingface-cli download Teensy/teensy-0 --local-dir ./teensy-0

Repository Layout

teensy-0/
├── checkpoints/       # trained PyTorch checkpoints
├── exported/          # HuggingFace-compatible safetensors + config.json
├── model.py           # TeensyLM architecture
├── sample.py          # text generation script for .pt
├── repl.py            # interactive streaming REPL for .pt
├── sample_hf.py       # text generation script for the safetensors export
├── repl_hf.py         # interactive streaming REPL for the safetensors export
├── export_hf.py       # export checkpoint to safetensors
├── README.md
├── LICENSE
└── requirements.txt

../colab/ — training & distillation toolkit (not part of this repo)

An experimental pipeline for the next Teensy variants (instruction tuning and tool calling) lives in the sibling colab/ directory, outside this repository, so the published model repo stays clean:

The original data, notebooks, and training scripts are archived in teensy-1/.

Limitations

• This model is not instruction-tuned.
• It performs next-token prediction only and will hallucinate or drift on open-ended prompts.
• It has no tool-calling or chat-following capability without further fine-tuning.
• Output quality is representative of a ~12M-parameter model trained on raw web text.
• Maximum context length is 64 tokens.

Credits and License

Copyright (c) 2025 Pankaj Doharey. Released under the MIT License.

Modified from NanoGPT by Andrej Karpathy.