Downloads · 30 days
0
0% of all-time downloads
Mistral-7B-Instruct-v0.1-qlora-polish
Azurro/Mistral-7B-Instruct-v0.1-qlora-polish
Mistral-7B-Instruct-v0.1-qlora-polish is a machine learning model from Azurro. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
This repo contains a qlora adapter for Mistral-7B-Instruct-v0.1, fine-tuned on a private instructions dataset, exclusively in Polish.
All-time downloads
7
Public
Repo size
336 MB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.bin336 MB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Oct 4, 2023
- Updated
- Oct 4, 2023
- SHA
- a14b29e2
Model card
From the Hugging Face model README
This repo contains a qlora adapter for Mistral-7B-Instruct-v0.1, fine-tuned on a private instructions dataset, exclusively in Polish.
The fine-tuning took 1 hour on a single RTX 4090 with the following hyperparameters:
- context length: 2048
- batch_size: 16
- learning_rate: 0.0001
- lora_r: 64
- lora_alpha: 16
- lora_modules: all
- lora_dropout: 0.0
- weight_decay: 0.1
- max_grad_norm: 0.3
- double_quant, nf4
- optimizer: paged_adamw_32bit (beta2: 0.999)
During the training of the adapter, the instruction format was maintained. Each prompt should be surrounded by [INST] and [\INST] tokens to leverage instruction fine-tuning.
This adapter allows the model to speak Polish more accurately than vanilla Mistral-7B-Instruct-v0.1.
<p align="center"> <img src="https://huggingface.co/Azurro/Mistral-7B-Instruct-v0.1-qlora-polish/raw/main/mistral-7b-instruct-qlora-pl.jpg"> </p>- License
- apache-2.0
- Languages
- pl
- Created
- Oct 4, 2023
- Updated
- Oct 4, 2023