booper-story-v1

booper-story-v1

Long-form SFT of ProCreations/Booper-Big-Chat-INT8 (Mixtral MoE, 150M total / ~50M active, vocab 16384) so booper answers story/long-answer requests instead of one-liners. Trained with sft/sft_longform.py from https://github.com/frgmt0/babble in the pair layout <bos> prompt <sep> response <eos> (loss on the response).

Data mix

• tinystories: 54000 examples
• no_robots: 8387 examples
• discord: 48000 examples

Training

• steps: 910, tokens: 27,998,826
• val loss: 2.4375 -> 1.3351
• device: mps, lr 4e-05, seq len 1024

Samples (temperature 0.8)

write me a short story about a dragon who is afraid of fire

Once upon a time there was a dragon called Tim. He wanted to be afraid of fire when he saw the dragon fluttering around. The dragon thought for a moment and then ran up to his friend. She said, "I'm afraid, but you have to go home first. Let's move on from the fearful dragon

tell me a story

I got upset at the beginning of the morning for a while and it started to rain outside.

Soon it was time for the rain. The rain stopped and it was all gone. I was so sad

hey booper whats up

My guy and me both don’t know who I am

Loading

Same INT8 layout as the base: load_int8.py in this repo, or babble.hfserve with BABBLE_HF_MODEL_DIR pointed at a snapshot.