Downloads · 30 days
195
6% of all-time downloads
RichardErkhov/AndreasThinks_-_mistral-7b-english-welsh-translate-gguf
AndreasThinks_-_mistral-7b-english-welsh-translate-gguf is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Downloads · 30 days
195
6% of all-time downloads
All-time downloads
3K
Public
Repo size
82.3 GB
Likes
0
Public
Click a slice to open those files.
.gguf95.3 GB · 100%
From the Hugging Face model README
Quantization made by Richard Erkhov.
mistral-7b-english-welsh-translate - GGUF
license: apache-2.0 base_model: mistralai/Mistral-7B-Instruct-v0.3 tags:
This model is a fine-tuned version of mistralai/Mistral-7B-Instruct-v0.3 on the Welsh Government Alpaca Welsh-English Instructions dataset.
This model is trained for English-Welsh translation (in any direction), with a focus on government documents, using Markdown formatting.
To ensure the highest quality translations, use the Alpaca instruction prompt format with the below structure.
### Instruction: {instruction}
### Input: {input}
### Response:
Your instruction should be "Translate the text from English to Welsh." (or vice versa).
The model is also available quantized as GGUF. This version be be tested in this interactive space.
The model is intended to be run locally, ideally using Text generation web UI to ensure correct prompt structure.
Start the UI as instructed for your system.
Performance may start to degrade past a certain context window (especially if using the quantized models). Convert in chunks of under 1000 words to avoid these issues.
Thanks to YALL - Yet Another LLM Leaderboard. Full results available here.
| Model | AGIEval | TruthfulQA | Bigbench |
|---|---|---|---|
| mistral-7b-english-welsh-translate | 35.31 | 54.5 | 38.4 |
axolotl version: 0.4.1
base_model: mistralai/Mistral-7B-Instruct-v0.3
model_type: MistralForCausalLM
tokenizer_type: LlamaTokenizer
load_in_8bit: false
load_in_4bit:
strict: false
# huggingface repo
datasets:
- path: AndreasThinks/welsh-translation-instruction
type: alpaca
val_set_size: 0.04
output_dir: ./outputs/mistral-welsh
hub_model_id: AndreasThinks/mistral-7b-english-welsh-translate
sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true
eval_sample_packing: false
wandb_project: mistral-nemo-welsh
wandb_entity:
wandb_watch:
wandb_name: mistral-nemo-welsh-v1
wandb_log_model:
gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 4
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.000005
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
warmup_steps: 10
evals_per_epoch: 4
eval_table_size:
eval_max_new_tokens: 128
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
</details><br>
The following hyperparameters were used during training:
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.5781 | 0.0013 | 1 | 1.5514 |
| 0.4427 | 0.2506 | 194 | 0.4841 |
| 0.4142 | 0.5011 | 388 | 0.4271 |
| 0.4001 | 0.7517 | 582 | 0.3996 |
| 0.4155 | 1.0023 | 776 | 0.3828 |
| 0.3178 | 1.2296 | 970 | 0.3792 |
| 0.3156 | 1.4801 | 1164 | 0.3732 |
| 0.3115 | 1.7307 | 1358 | 0.3678 |
| 0.2722 | 1.9813 | 1552 | 0.3633 |
| 0.2492 | 2.2089 | 1746 | 0.3809 |
| 0.2159 | 2.4595 | 1940 | 0.3828 |
| 0.2277 | 2.7100 | 2134 | 0.3810 |
| 0.2435 | 2.9606 | 2328 | 0.3809 |
| 0.158 | 3.1899 | 2522 | 0.3961 |
| 0.1802 | 3.4404 | 2716 | 0.3966 |
| 0.1679 | 3.6910 | 2910 | 0.3966 |