Downloads · 30 days
11
4% of all-time downloads
BigMistral-13b-GLUED
athirdpath/BigMistral-13b-GLUED
BigMistral-13b-GLUED is a text generation model from athirdpath. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
This is BigMistral-13b (a 13b Mistral base model, using a modified NeverSleep recipe) merged with a 512-rank LORA trained over it directly.
All-time downloads
268
Public
Repo size
53.4 GB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.bin26.7 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Model type
- mistral
- Access
- Public
- Created
- Dec 3, 2023
- Updated
- Dec 3, 2023
- SHA
- 25d850ae
Try a prompt
Model card
From the Hugging Face model README
Description
This is BigMistral-13b (a 13b Mistral base model, using a modified NeverSleep recipe) merged with a 512-rank LORA trained over it directly.
Logic
My 20b Llama 2 merges did well, in part due to the inclusion of Elieithyia-20b, trained on top of a 20b merge directly. This time, I trained the LoRA not with the traditional goals in mind, but with “healing” the 13b merge. This involved increasing the gradient accumulation steps significantly, lowering the learning rate, and decreasing the dropout. Fingers crossed!
Results
Prelimary results are an improvement. Really needs a bigger dataset of more cognitive tasks like OpenOrca. Writes okay stories and responds mostly factually. Idea is solid.
Dataset
This LORA was trained on a dataset that consists of teknium1’s roleplay-instruct-v2.1, and then part of the private Elieithyia dataset and of HF’s No Robots, chosen randomly to form a even (filesize) 3 way split.
- Task
- Text Generation
- Library
- transformers
- Type
- mistral
- License
- apache-2.0
- Created
- Dec 3, 2023
- Updated
- Dec 3, 2023