roast-bot-qwen-1b

Roast Bot — Qwen 1.5B (full training run)

Fine-tuned Qwen2.5-1.5B-Instruct, trained on Sarvin's WhatsApp texting style. LoRA r=32, 5 epochs, 90 example conversation pairs.

Looking for a version that generalizes better instead of reciting training data? See roast-bot-qwen-1b-epoch2 — an earlier checkpoint from the same run that trades a bit of surface polish for actually synthesizing responses instead of memorizing them. Read the comparison below before picking one.

Which checkpoint should I use?

This repo is the epoch-5 (fully trained) checkpoint — the one that finished all 5 epochs. On a held-out eval split, training loss kept falling every epoch but eval loss bottomed out around epoch 1-2 and rose afterward — the textbook signature of a model starting to memorize its training set rather than learn the general pattern behind it.

In practice, this shows up as: inputs that closely resemble one of the 90 training examples often get an exact, verbatim, word-for-word reply — not a new response, a lookup. On inputs unlike anything in training, quality is more inconsistent than epoch2.

Use this checkpoint if: you mostly care about polish on inputs similar to the original 90 training examples, and don't mind that similarity sometimes means an exact copy of training data rather than a generated response.

Use epoch2 instead if: you want a model that's actually generating responses to what you send it, even if individual lines are occasionally a little rougher.

Training details

• Base: unsloth/Qwen2.5-1.5B-Instruct-bnb-4bit
• Method: LoRA (r=32, alpha=64, dropout=0.05) via Unsloth
• Data: 90 hand-written conversation pairs (80 train / 10 eval split, no duplication)
• 5 epochs, lr=2e-4, cosine schedule, weight decay 0.01
• Eval loss by epoch: 0.652 → 0.661 → 0.930 → 0.946 → 0.989 (this checkpoint)
• Training notebook: training/roast_bot_colab.ipynb

Usage

Deployed via Ollama in roast-bot, a WhatsApp bot. See that repo for the full inference pipeline (system prompt, reply-decision logic, GGUF conversion steps).