Caelistus-Gladius-Excalibur-RP-2.6B-Thinking-v0.1

Caelistus-Gladius-Excalibur-RP-2.6B-Thinking-v0.1: experimental rank 2048 DoRA finetune of LFM 2.5 2.6B using beyoru/Aesir-Character-CoT-roleplay dataset, pure madness.

"Whoso Pulleth Out This Sword From This Stone and Anvil is Rightwise King Born of England"

The meaning of the experiment:

I wanted to completely reshape the way model thinks, so I took on a little special approach, combining DoRA with ridiculously high rank(2048).

• What is DoRA? DoRA is a weight decomposed version of LoRA that achieves higher accuracy at the cost of more training compute, this repo contains merged model, meaning that overhead for the end user is zero.

Here is some horrible photographic evidence:

Final step loss, if anyone is interested, was around 0.95-1~ for this version.

Special thanks:

To Beyoru and Gryphe for the amazing dataset.

And to Unsloth for convenient training.

Note:

Quants are located in the repo.

Training details:

• Method: DoRA
• Rank: 2048
• Training hardware: 5070 ti
• Trained for: 5~ hours