Downloads · 30 days
19
7% of all-time downloads
Dendrite-8x7Bv1
Envoid/Dendrite-8x7Bv1
Dendrite-8x7Bv1 is a text generation model from Envoid. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as cc-by-nc-4.0.
This is my first attempt at a qLoRA on Mixtral using the same raw text dataset as the Dendrite model on Mixtral 8x7B Instruct v0.1 with the following notable settings:
All-time downloads
259
Public
Parameters
46.7B
93.4 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors93.4 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- cc-by-nc-4.0
- Model type
- mixtral
- Access
- Public
- Created
- Jan 3, 2024
- Updated
- Jan 3, 2024
- SHA
- de3c9190
Try a prompt
Model card
From the Hugging Face model README
This is my first attempt at a qLoRA on Mixtral using the same raw text dataset as the Dendrite model on Mixtral 8x7B Instruct v0.1 with the following notable settings:
lora_r 256
lora_alpha 256
..._max_len 256
learning_rate 0.000001
num_train_epochs 2
The output is a little less dry but it still maintains the full level of functionality you would expect out of Mixtral instruct. And it still responds to the
[INST]
Do a thing
[/INST]
format. It's not a massive change in the output but I do plan to run a similar but larger data set with more epochs and a higher learning rate.
- Task
- Text Generation
- Library
- transformers
- Type
- mixtral
- License
- cc-by-nc-4.0
- Created
- Jan 3, 2024
- Updated
- Jan 3, 2024