NEBULA-23.8B-v1.0

NEBULA-23.8B-v1.0

Technical notes

• 108 layers,DUS procedure, mistral(32)->SOLAR(48)->GALAXY(72)->NEBULA(108)
• 23.8B parameters
• model created as an extension of depth upscaling procedure used for SOLAR by upstage

Results

• model can and will produce NSFW content
• GSM8k evaluation seems to be often broken, HellaSwag, Winograde and TQA show that its a smart model
• RP and ERP work surprisingly good and I didn't encounter any GPTisms yet
• comparable memory footprint to 20B and 23B models based on llama
• follows character card very well
• NSFW output feels fresh comparing to existing models

Finetuning for RP

• SFT using MinervaAI/Aesir-Preview dataset, 10 epochs
• DPO using athirdpath/DPO_Pairs-Roleplay-Alpaca-NSFW dataset, 1 epoch
• SFT using 1xAda6000, 10h
• DPO using 1x3090, 30h
• jupyter notebooks or mergekit configs for anyone wanting to reproduce/reuse scripts - just drop me a message

Prompt template

• Alpaca
• chat template is embedded in tokenizer config, should load automatically

Context size

• 4096

All comments are greatly appreciated, download, test and if you appreciate my work, consider buying me my fuel: <a href="https://www.buymeacoffee.com/TeeZee" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" style="height: 60px !important;width: 217px !important;" ></a>

Open LLM Leaderboard Evaluation Results

Detailed results can be found here