Downloads · 30 days
10
2% of all-time downloads
Airoboros-13b-SuperHOT-8k
Peeepy/Airoboros-13b-SuperHOT-8k
Airoboros-13b-SuperHOT-8k is a text generation model from Peeepy. Use it when you need the model to write or continue text. It is set up for transformers.
Airoboros 13b GPT4 1.4 merged with kaiokendev's SuperHOT 8k LoRA.
All-time downloads
593
Public
Repo size
52.1 GB
Likes
2
Public
Repo makeup
Click a slice to open those files.
.bin26 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- Model type
- llama
- Access
- Public
- Created
- Jun 22, 2023
- Updated
- Jun 23, 2023
- SHA
- b68311ba
Try a prompt
Model card
From the Hugging Face model README
Airoboros 13b GPT4 1.4 merged with kaiokendev's SuperHOT 8k LoRA.
The code to merge these can be found here. Change information as needed.
NOTE: This requires a monkey patch to work. FlashVenom has, along with kindly quantising this model to 4bit, added the monkeypatch file to their repo. You can access this here.
FROM THE ORIGINAL LORA MODEL CARD: This is a second prototype of SuperHOT, this time with 4K context and no RLHF. In my testing, it can go all the way to 6K without breaking down and I made the change with intention to reach 8K, so I'll assume it will go to 8K although I only trained on 4K sequences.
In order to use the 8K context, you will need to apply the monkeypatch I have added in this repo -- without it, it will not work. The patch is very simple, and you can make the changes yourself:
Increase the max_position_embeddings to 8192 to stretch the sinusoidal
Stretch the frequency steps by a scale of 0.25
- Task
- Text Generation
- Library
- transformers
- Type
- llama
- Created
- Jun 22, 2023
- Updated
- Jun 23, 2023