llama-30b-SuperHOT-4bit

Merge of SuperHOT-LoRA-prototype and llama-30b

Llama30B-SuperHOT-4bit-128g.safetensors Quantization:

CUDA_VISIBLE_DEVICES=0 python llama.py ausboss/Llama30B-SuperHOT c4 --wbits 4 --true-sequential --groupsize 128 --save_safetensors Llama30B-SuperHOT-4bit-128g.safetensors

Llama30B-SuperHOT-4bit.safetensors Quantization:

CUDA_VISIBLE_DEVICES=0 python llama.py ausboss/Llama30B-SuperHOT c4 --wbits 4 --true-sequential --save_safetensors Llama30B-SuperHOT-4bit.safetensors

From the SuperHot Page:

Prototypes for SuperHOT

No guarantees for output quality, simply uploading what I have so others can play around with it. Not even sure if the rank in cutoff-8192 is correct (think it should be 10 maybe.. can't remember)

All prototypes are extremely early epochs (sub 0.5)

Model/Training

All trained with Flash Attention with conversation sequence lengths ranging from 8K to 16K tokens (No Alibi unless otherwise mentioned)

All trained on LLaMa 13B 4-bit (no groupsize)

(Personally, I like the 8K cutoff version better, so I would say start with that one)

Data

A combination of various datasets and cleaned logs converted into datasets including but not limited to:

• Bluemoon Fanbased
• Roleplaying Guild
• Community-sourced outputs
• Dan's PocketDoc/RUCAIBox-Story-Generation-Alpaca
• IlyaGusev/gpt_roleplay_realm
• others

Bias

SuperHOT is a fiction-focused model. No alignment has been performed on the training data. Be mindful that this model may output harmful, violent, or otherwise problematic content

Format

Any format should work with such early checkpoints. However the training data is entirely in the following format:

---
mode: chat
characters:
    <char1 name>: <descriptive tags for char1>
    <char2 name>: <descriptive tags for char2>
summary: <summary of the story thus far or the purpose of the chat> (optional)
<any other miscellaneous data>
---
<chat history>

By "any other miscellaneous data", it means you should be able to put any additional metadata for the story or characters. I.e.,

...
locations:
    location1: <tags for location1>
inventory:
    item1: <tags for item1>

Again, format does not hold such a large weight on these early checkpoints. I have found success with the following setup for an RPG-like experience. Just play around with the format and see what works:

---
mode: rpg
characters:
     You: a new player
system: The system controls the RPG, handles character creation, world narration, and quest management. Also controls any NPCs and inventory tracking. Their first message provides a lengthy introduction for the player into the RPG world they are about to play in. After completing the character creation, the system will give a lengthy introduction into the world of ___. The first quest will follow right after
rpg setting: The world of ___
rpg rules: Any rules typical of RPG games, including typical items, battle stats, etc
---