kevin-chat

KEVIN-chat

A decoder-only transformer built entirely from scratch — every forward and backward pass derived and coded by hand, with no autograd, no PyTorch, no HuggingFace transformers. Implemented twice: once in NumPy, once in CUDA/C++ (this checkpoint is from the CUDA trainer).

This is the chat/instruction-tuned variant: SFT'd from kevinindustries/kevin (the base model) on conversation-format data.

Architecture

Identical to the base model — a LLaMA-style pre-norm decoder stack:

• 82.9M parameters
• d_model = 512
• 12 layers, 8 attention heads (d_head = 64)
• SwiGLU feed-forward (d_ff = 2026), with a learned Swish-gate β
• RMSNorm instead of LayerNorm
• Learned positional embeddings (no RoPE)
• Context window: 256 tokens, no KV cache — every generated token is a full forward pass over the whole context
• Untied input/output embeddings
• Vocab size: 32,005 (32,000-token byte-level BPE + 5 special tokens, one of which, <|endoftext|>, is a real single-token end-of-turn signal - see Training below)

Training

• SFT'd from the base model (kevinindustries/kevin, step 698,000) on HuggingFaceTB/everyday-conversations-llama3.1-2k (2,379 short everyday-conversation exchanges), chat/conversation format (<|system|>/<|user|>/<|assistant|> turns, rendered as ordinary BPE text, not registered special tokens)
• Optimizer: AdamW, peak LR 5e-5, cosine decay to 5e-6, 100-500 warmup steps (varies by restart), global gradient-norm clipping, label smoothing 0.05, dropout 0.1
• Loss is computed only on assistant-turn tokens
• This checkpoint: step 162,000, loss ~0.73-0.85, perplexity ~2.1-2.3.

Known limitations of this checkpoint

• The training loop has no epoch tracking - it draws random 256-token windows from the corpus with replacement, indefinitely, rather than iterating the dataset once per epoch. At step 162,000 (batch size 12, 256-token context) the model has been sampled roughly ~650 times over the ~760K-token corpus. It shows: for common inputs (e.g. "Hi") it tends to reproduce a memorized reply verbatim (e.g. "Hello! How can I help you today?") rather than a fresh completion. Treat this as a heavily overfit checkpoint on a very small SFT set, not a well-generalized chat model. An earlier checkpoint from the same run (ckpt_step500.ckpt, ~2 passes over the corpus) is much closer to a single epoch and generalizes better, at the cost of not yet reliably ending its turn (see next point) - ask in the project repo if you want that checkpoint instead.
• <|endoftext|> end-of-turn training only covers the last ~17,000 of these 162,000 steps. Earlier SFT data rendering never appended an explicit end-of-turn token; the model instead learned to signal "my turn is over" by spelling out the literal text of the next role tag (e.g. <|user|>), which is fragile - one wrong token in that sequence and generation runs on. <|endoftext|> is now appended after every assistant turn during training (one of the tokenizer's previously-unused reserved special ids), giving the model a single-token way to stop, but it's had comparatively little exposure to it so far and turns don't always end cleanly.
• Inference-side code (see the project repo's serve.py/utils/generate.py) also supports a CTRL-style repetition penalty (penalize logits of tokens already in context) as a mitigation for the verbatim-repeat behavior above; it's implemented but off by default (repetition_penalty=1.0) since it can distort otherwise-coherent continuations, and it doesn't address the root cause (memorization from over-sampling a tiny dataset).

Checkpoint format

latest.ckpt is the CUDA trainer's native binary format (TFCKPT1 magic header), not a PyTorch state_dict. It carries its own architecture header (step, vocab size, d_model, heads, layers, d_ff, max_len) so it's self-describing. See the project repo for the loader (utils/ckpt_convert.py, cuda/include/checkpoint.cuh) and inference code (generate.py, serve.py).

Repo contents

• latest.ckpt — the checkpoint itself.
• tokenizer/tokenizer.bbpe (+ merges.txt, vocab.json) — the from-scratch byte-level BPE tokenizer this checkpoint was trained with (same tokenizer as the base model). Must match exactly; a different tokenizer/vocab size will silently produce garbage.

Download with:

hf download kevinindustries/kevin-chat --local-dir kevin-chat

Then resume SFT training (note: consider fixing the epoch-tracking / overfitting issue above - e.g. training a fresh run from the base model for a bounded, small step count - before continuing this particular lineage further):

./build/train_transformer_cuda \
  --resume kevin-chat/latest.ckpt \
  --corpus <your-chat-corpus.jsonl> --data-format chat \
  --tokenizer bbpe --tokenizer-path kevin-chat/tokenizer/tokenizer.bbpe \
  --batch-size 12 --lr 5e-5 --min-lr 5e-6 --steps 3000 --warmup-steps 100 \
  --grad-clip 1.0 --label-smoothing 0.05 --dropout 0.1 \
  --log-every 50 --checkpoint-every 500 \
  --checkpoint-dir checkpoints --metrics-path checkpoints/metrics.csv

The base (non-chat) model is at kevinindustries/kevin.

Project repo: https://github.com/pstaykov/transformer