zephyr-7b-sft-lora-accum4-lr1e_6

shkang/zephyr-7b-sft-lora-accum4-lr1e_6

zephyr-7b-sft-lora-accum4-lr1e_6 is a machine learning model from shkang. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.

should probably proofread and complete it, then remove this comment. --

Downloads · 30 days

Access

Public

Updated Dec 10, 2023

Repo size

109 MB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors109 MB · 98%

At a glance

License: apache-2.0
Access: Public
Created: Dec 10, 2023
Updated: Dec 10, 2023
SHA: d94cd66b

Base models

mistralai/Mistral-7B-v0.1

Model card

From the Hugging Face model README

zephyr-7b-sft-lora-accum4-lr1e_6

This model is a fine-tuned version of mistralai/Mistral-7B-v0.1 on an unknown dataset. It achieves the following results on the evaluation set:

Loss: 1.6318

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 1e-06
train_batch_size: 4
eval_batch_size: 8
seed: 42
distributed_type: multi-GPU
num_devices: 2
gradient_accumulation_steps: 4
total_train_batch_size: 32
total_eval_batch_size: 16
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
num_epochs: 50.0

Training results

Training Loss	Epoch	Step	Validation Loss
2.0813	0.55	13	2.0600
2.0806	1.57	27	2.0523
2.0639	2.55	40	2.0440
2.0529	3.57	54	2.0326
2.0351	4.55	67	2.0158
2.0181	5.57	81	1.9973
2.0016	6.55	94	1.9805
1.9885	7.57	108	1.9627
1.9667	8.55	121	1.9449
1.9513	9.57	135	1.9297
1.9428	10.55	148	1.9160
1.9284	11.57	162	1.8975
1.9081	12.55	175	1.8844
1.8966	13.57	189	1.8704
1.8716	14.55	202	1.8590
1.8685	15.57	216	1.8439
1.8549	16.55	229	1.8304
1.8454	17.57	243	1.8203
1.839	18.55	256	1.8097
1.8293	19.57	270	1.7974
1.8162	20.55	283	1.7916
1.8076	21.57	297	1.7809
1.7915	22.55	310	1.7738
1.7749	23.57	324	1.7648
1.7907	24.55	337	1.7553
1.7633	25.57	351	1.7481
1.7554	26.55	364	1.7411
1.7604	27.57	378	1.7348
1.7497	28.55	391	1.7285
1.7574	29.57	405	1.7214
1.729	30.55	418	1.7154
1.7397	31.57	432	1.7102
1.7211	32.55	445	1.7026
1.7275	33.57	459	1.6945
1.709	34.55	472	1.6913
1.6983	35.57	486	1.6878
1.7056	36.55	499	1.6837
1.6996	37.57	513	1.6772
1.6929	38.55	526	1.6753
1.681	39.57	540	1.6701
1.6974	40.55	553	1.6647
1.6892	41.57	567	1.6599
1.6878	42.55	580	1.6538
1.6745	43.57	594	1.6531
1.6555	44.55	607	1.6464
1.6643	45.57	621	1.6449
1.6441	46.55	634	1.6414
1.6572	47.57	648	1.6362
1.6461	48.55	661	1.6343
1.6605	49.57	675	1.6313

Framework versions

Transformers 4.35.0
Pytorch 2.1.0
Datasets 2.14.6
Tokenizers 0.14.1

License: apache-2.0
Created: Dec 10, 2023
Updated: Dec 10, 2023