zephyr-7b-sft-lora-accum8-lr3e_6

shkang/zephyr-7b-sft-lora-accum8-lr3e_6

zephyr-7b-sft-lora-accum8-lr3e_6 is a machine learning model from shkang. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.

should probably proofread and complete it, then remove this comment. --

Downloads · 30 days

Access

Public

Updated Dec 10, 2023

Repo size

109 MB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors109 MB · 98%

At a glance

License: apache-2.0
Access: Public
Created: Dec 10, 2023
Updated: Dec 10, 2023
SHA: 20f6c09b

Base models

mistralai/Mistral-7B-v0.1

Model card

From the Hugging Face model README

zephyr-7b-sft-lora-accum8-lr3e_6

This model is a fine-tuned version of mistralai/Mistral-7B-v0.1 on an unknown dataset. It achieves the following results on the evaluation set:

Loss: 1.4622

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 3e-06
train_batch_size: 4
eval_batch_size: 8
seed: 42
distributed_type: multi-GPU
num_devices: 2
gradient_accumulation_steps: 8
total_train_batch_size: 64
total_eval_batch_size: 16
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
num_epochs: 50.0

Training results

Training Loss	Epoch	Step	Validation Loss
2.0845	0.51	6	2.0584
2.068	1.53	13	2.0440
2.0512	2.55	20	2.0264
2.0319	3.57	27	2.0058
2.0251	4.51	33	1.9782
1.9819	5.53	40	1.9558
1.9596	6.55	47	1.9264
1.9494	7.57	54	1.9030
1.9132	8.51	60	1.8867
1.8962	9.53	67	1.8628
1.8742	10.55	74	1.8444
1.8577	11.57	81	1.8221
1.8321	12.51	87	1.8100
1.8306	13.53	94	1.7924
1.7979	14.55	101	1.7773
1.791	15.57	108	1.7632
1.7883	16.51	114	1.7495
1.7608	17.53	121	1.7364
1.75	18.55	128	1.7245
1.7351	19.57	135	1.7115
1.7265	20.51	141	1.7055
1.7191	21.53	148	1.6928
1.6983	22.55	155	1.6830
1.6958	23.57	162	1.6723
1.6862	24.51	168	1.6620
1.6699	25.53	175	1.6528
1.6718	26.55	182	1.6424
1.6637	27.57	189	1.6338
1.6483	28.51	195	1.6256
1.6333	29.53	202	1.6159
1.6415	30.55	209	1.6068
1.6212	31.57	216	1.5987
1.6196	32.51	222	1.5890
1.595	33.53	229	1.5779
1.596	34.55	236	1.5714
1.5919	35.57	243	1.5647
1.5814	36.51	249	1.5579
1.574	37.53	256	1.5486
1.5621	38.55	263	1.5426
1.5476	39.57	270	1.5344
1.5469	40.51	276	1.5263
1.5418	41.53	283	1.5177
1.5305	42.55	290	1.5082
1.523	43.57	297	1.5043
1.5282	44.51	303	1.4947
1.5095	45.53	310	1.4895
1.4989	46.55	317	1.4816
1.5023	47.57	324	1.4734
1.491	48.51	330	1.4687
1.4814	49.53	337	1.4620

Framework versions

Transformers 4.35.0
Pytorch 2.1.0
Datasets 2.14.6
Tokenizers 0.14.1

License: apache-2.0
Created: Dec 10, 2023
Updated: Dec 10, 2023