zephyr-7b-sft-lora-accum4-lr1e_5

shkang/zephyr-7b-sft-lora-accum4-lr1e_5

zephyr-7b-sft-lora-accum4-lr1e_5 is a machine learning model from shkang. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.

should probably proofread and complete it, then remove this comment. --

Downloads · 30 days

Access

Public

Updated Dec 10, 2023

Repo size

109 MB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors109 MB · 98%

At a glance

License: apache-2.0
Access: Public
Created: Dec 10, 2023
Updated: Dec 10, 2023
SHA: 56c1bb19

Base models

mistralai/Mistral-7B-v0.1

Model card

From the Hugging Face model README

zephyr-7b-sft-lora-accum4-lr1e_5

This model is a fine-tuned version of mistralai/Mistral-7B-v0.1 on an unknown dataset. It achieves the following results on the evaluation set:

Loss: 0.7880

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 1e-05
train_batch_size: 4
eval_batch_size: 8
seed: 42
distributed_type: multi-GPU
num_devices: 2
gradient_accumulation_steps: 4
total_train_batch_size: 32
total_eval_batch_size: 16
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
num_epochs: 50.0

Training results

Training Loss	Epoch	Step	Validation Loss
2.041	0.55	13	1.9551
1.8921	1.57	27	1.8206
1.7735	2.55	40	1.7331
1.7122	3.57	54	1.6613
1.63	4.55	67	1.5922
1.5552	5.57	81	1.5170
1.4953	6.55	94	1.4338
1.3904	7.57	108	1.3306
1.2882	8.55	121	1.2588
1.2282	9.57	135	1.2057
1.2016	10.55	148	1.1698
1.1707	11.57	162	1.1430
1.1417	12.55	175	1.1254
1.1281	13.57	189	1.1089
1.1005	14.55	202	1.0957
1.101	15.57	216	1.0832
1.0825	16.55	229	1.0708
1.0743	17.57	243	1.0605
1.0642	18.55	256	1.0508
1.0486	19.57	270	1.0392
1.0383	20.55	283	1.0343
1.0281	21.57	297	1.0229
1.0129	22.55	310	1.0152
0.9941	23.57	324	1.0072
1.0026	24.55	337	0.9972
0.9769	25.57	351	0.9888
0.9641	26.55	364	0.9800
0.9636	27.57	378	0.9725
0.9503	28.55	391	0.9626
0.9463	29.57	405	0.9541
0.9194	30.55	418	0.9462
0.9213	31.57	432	0.9374
0.8971	32.55	445	0.9283
0.8972	33.57	459	0.9160
0.872	34.55	472	0.9080
0.8633	35.57	486	0.9000
0.8565	36.55	499	0.8920
0.8479	37.57	513	0.8804
0.8332	38.55	526	0.8737
0.8216	39.57	540	0.8650
0.8112	40.55	553	0.8568
0.7988	41.57	567	0.8463
0.7862	42.55	580	0.8370
0.7675	43.57	594	0.8311
0.761	44.55	607	0.8224
0.757	45.57	621	0.8158
0.7426	46.55	634	0.8089
0.7303	47.57	648	0.8007
0.7169	48.55	661	0.7946
0.7249	49.57	675	0.7884

Framework versions

Transformers 4.35.0
Pytorch 2.1.0
Datasets 2.14.6
Tokenizers 0.14.1

License: apache-2.0
Created: Dec 10, 2023
Updated: Dec 10, 2023