Tlacuilo-12B

I made another creating writing model.

But! Unlike the last couple! This one actually doesn't suck at RP or adventure! Hooray!

Usage notes: Chat template is ChatML, since this was trained using Muse-12B as the base model. Works with Temperature 1 / min-p 0.05, up through about Temperature 1.3 / min-p 0.02 if you like it hotter, probably works with other settings too.

Training notes: I started with Muse-12B because I like, in general, how this model behaves for creative uses. It's been my daily driver for a while, but I felt like the prose could stand to be a little more variable.

The training process went like this:

Stage 1:

• Data: A bunch of books (chosen for their prose/writing style). About 28M tokens per epoch.
• r32/a32 QLoRA at 32k context, applied only to the QKV tensors. LR: 1e-5, 2 epochs.

Stage 2:

• Trained on top of Stage 1.
• Data: RP data, about 4M tokens.
• r32/a32 QLoRA at 16k context, applied to o_proj and down_proj only. LR: 5e-6, 1 epoch.

Stage 3:

• Trained on top of Stage 2.
• Data: Instruct data (1000 random samples from koto-instruct-sft), about 1.2M tokens.
• r32/a32 QLoRA at 4k context, applied to all linear modules. LR: 2e-6, 1 epoch.

Why?

Dunno. Fun experiment. Seemed to work, at least.

Enjoy!

(Big thanks to Gryphe and LatitudeGames for making Muse-12B. Great model, knocked it out of the park fr)