Downloads · 30 days
9
1% of all-time downloads
AusLegalQA-Mixtral-8x7B-Instruct-v0.1
adlumal/AusLegalQA-Mixtral-8x7B-Instruct-v0.1
AusLegalQA-Mixtral-8x7B-Instruct-v0.1 is a text generation model from adlumal. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
AusLegalQA is a fine-tune of Mistral-8x7B-Instruct-0.1 using PEFT techniques, trained on the Open Australian Legal QA.
All-time downloads
642
Public
Parameters
46.7B
93.4 GB on disk
Likes
3
Public
Repo makeup
Click a slice to open those files.
.safetensors93.4 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Model type
- mixtral
- Access
- Public
- Created
- Dec 22, 2023
- Updated
- Dec 23, 2023
- SHA
- 419eba86
Try a prompt
Model card
From the Hugging Face model README
AusLegalQA
AusLegalQA is a fine-tune of Mistral-8x7B-Instruct-0.1 using PEFT techniques, trained on the Open Australian Legal QA.
The model achieved an eval loss of 1.1391 on a subset of 100 prompts and answers from the original dataset.
The model was trained with the following hyperparameters for 3 epochs. The step with the lowest eval loss was selected (coinciding with end of epoch 2) and the resulting qLoRA (4 bits) was merged into the base model.
| Hyperparameter | Value |
|---|---|
| Sequence length | 1024 |
| Epochs | 2 |
| Optimiser | AdamW |
| Learning rate | 1e-4 |
| Learning rate scheduler | Cosine |
| Batch size | 1 |
| Weight decay | 0.01 |
| Warmup ratio | 0.05 |
| LoRA rank | 64 |
| LoRA alpha | 128 |
| LoRA dropout | 0.1 |
| LoRA target | q_proj,v_proj |
| NEFTune alpha | 5 |
| Flash Attention | on |
Strengths
The model is strong at summarisation and short-form answers with the key details. It is more likely to provide responses which assume the user is located in Australia. Ideal use-case is in a LLamaIndex/LangChain environment.
Limitations
Just as the base model it does not have any moderation mechanisms.
- Task
- Text Generation
- Library
- transformers
- Type
- mixtral
- License
- apache-2.0
- Created
- Dec 22, 2023
- Updated
- Dec 23, 2023