bdh-100m-bytes-chat-v3

BDH 100M, bytes, chat v3

bdh-100m-bytes tuned for 800 steps on 6,960 short conversation-shaped documents: story requests answered with a TinyStories-style story, stories followed by questions whose answers are only in the story, list requests followed by position questions ("what was the second one?"), and a small set of one-line tasks and stop markers. Loss was taken only on the assistant's replies. The tuning data was written by an LLM from category prompts and is not in this release.

Format: turns are User: ... and Assistant: ... on their own lines. A reply ends with <|endoftext|>. chat_stream.py in the code repository handles this and keeps the whole conversation in the model's synaptic state with a 480-position sliding window.

What tuning changed

• Recall of a fact planted earlier in the context (the planted-fact probe, 30 trials per distance): 97 % at 128 characters against 20 % for the base model, then 7 % at 256 and 13 % at 384. The recall depends on the key the fact was planted under. For instance, with the word planted as "the dog's name" and the question still asking for the secret word, recall at 128 characters falls from 29 of 30 to 11 of 30.
• On a 60-item multi-turn exam (stories with three questions each, lists with position questions), 16 % of its story answers were judged supported by its own story, and it answered 79 % of "which was first" questions correctly, but 7 % of "which was second".
• Held-out bits per character rose from 1.042 to 1.340 (+29 %): the tune makes it a worse general language model, because the tuning text was never mixed with pretraining text.

Training

800 steps of 32 sequences, AdamW with weight decay 0.01, learning rate 1e-4 flat, dropout 0.1, bf16, about four passes over the data. One RTX 4090, 25 minutes.

Use

TEMP=0.3 python chat_stream.py bdh-100m-bytes-chat-v3.pt

Limitations

A 100M-parameter model tuned on a few thousand short dialogues. It answers in the register of children's stories, invents facts ("Water boils at 100°F"), and chooses the most recent item in a list when it cannot recall. Its 480-position window holds a short story and one or two questions. By the third question much of the story has left the window. Single training seed for the base. The exam numbers are the mean of three generation seeds.

Code: https://github.com/d3vmeh/small-dragon-hatchling. Paper: https://doi.org/10.5281/zenodo.23325933 (CC BY 4.0).