Downloads · 30 days
6
10% of all-time downloads
FlatDolphinMaid-8x7B-3bpw-exl2
Kooten/FlatDolphinMaid-8x7B-3bpw-exl2
FlatDolphinMaid-8x7B-3bpw-exl2 is a text generation model from Kooten. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as cc-by-nc-4.0.
Exllama quant of Undi95/FlatDolphinMaid-8x7B
All-time downloads
59
Public
Repo size
17.8 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors17.8 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- cc-by-nc-4.0
- Model type
- mixtral
- Access
- Public
- Created
- Jan 3, 2024
- Updated
- Jan 3, 2024
- SHA
- f3f3033f
Try a prompt
Model card
From the Hugging Face model README
FlatDolphinMaid-8x7B 3bpw
Exllama quant of Undi95/FlatDolphinMaid-8x7B
You probably want the 3.5bpw version. It just fits in 24gb of vram at half context (16384).
If you really want the larger context 3bpw should do it but you are probably better of with the gguf version with higher quants.
I did make a 4bpw, it might work in a headless or multigpu setup.
Other BPW's 3.0bpw, 3.5bpw, 4.0bpw
Make sure you enable 8bit cache.
Promt format:
### Instruction:
{system prompt}
### Input:
{input}
### Response:
{reply}
Contact
Kooten on discord.
- Task
- Text Generation
- Library
- transformers
- Type
- mixtral
- License
- cc-by-nc-4.0
- Created
- Jan 3, 2024
- Updated
- Jan 3, 2024