booper-story-v2

booper-story-v2

Long-form SFT of ProCreations/Booper-Big-Chat-INT8 (Mixtral MoE, 150M total / ~50M active, vocab 16384) so booper answers story/long-answer requests instead of one-liners. Trained with sft/sft_longform.py from https://github.com/frgmt0/babble in the pair layout <bos> prompt <sep> response <eos> (loss on the response).

Data mix

• tinystories: 32000 examples
• writingprompts: 40000 examples
• no_robots: 16000 examples
• smoltalk: 16000 examples
• discord: 56000 examples

Training

• steps: 1220, tokens: 37,022,555
• val loss: 2.0340 -> 1.7995
• device: mps, lr 4e-05, seq len 1024

Samples (temperature 0.8)

write me a short story about a dragon who is afraid of fire

Once upon a time, there was a dragon who lived in the forest. He lived in a small and strong forest, where he had to be careful of trees. One day, the dragon heard a noise from his neighbors. He told the dragon that he wanted to play with his friends, but it was too late.

write a short story about a detective in a city that never sleeps

Once upon a time there was a very strong detective. It would be a very good night until it was time to go back to sleep. One day, the detective had been up for the night. He was getting ready to go. As he was looking around, he saw a big room in his right hand. The detective

hey booper whats up

i gotta go on a walk

Loading

Same INT8 layout as the base: load_int8.py in this repo, or babble.hfserve with BABBLE_HF_MODEL_DIR pointed at a snapshot.