mistral_7b_cosine_w_restart_lr

zeeshan73/mistral_7b_cosine_w_restart_lr

mistral_7b_cosine_w_restart_lr is a machine learning model from zeeshan73. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as apache-2.0.

should probably proofread and complete it, then remove this comment. --

Downloads · 30 days

68% of all-time downloads

All-time downloads

Public

Repo size

187 GB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors3.2 GB · 100%

At a glance

Library: peft
License: apache-2.0
Access: Public
Created: Oct 26, 2024
Updated: Oct 26, 2024
SHA: ecdc7976

Base models

mistralai/Mistral-7B-Instruct-v0.3

Datasets

generator

Model card

From the Hugging Face model README

mistral_7b_cosine_lr

This model is a fine-tuned version of mistralai/Mistral-7B-Instruct-v0.3 on the generator dataset. It achieves the following results on the evaluation set:

Loss: 0.3803

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0002
train_batch_size: 2
eval_batch_size: 8
seed: 42
gradient_accumulation_steps: 8
total_train_batch_size: 16
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine_with_restarts
lr_scheduler_warmup_ratio: 0.03
lr_scheduler_warmup_steps: 15
num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
1.0242	0.0366	10	0.6227
0.5884	0.0732	20	0.5310
0.519	0.1098	30	0.4930
0.4818	0.1465	40	0.4653
0.4722	0.1831	50	0.4537
0.4513	0.2197	60	0.4440
0.4481	0.2563	70	0.4377
0.4455	0.2929	80	0.4321
0.4344	0.3295	90	0.4271
0.4345	0.3661	100	0.4233
0.4296	0.4027	110	0.4186
0.4255	0.4394	120	0.4166
0.4173	0.4760	130	0.4131
0.4195	0.5126	140	0.4098
0.4143	0.5492	150	0.4067
0.4103	0.5858	160	0.4043
0.4124	0.6224	170	0.4021
0.4069	0.6590	180	0.3988
0.4041	0.6957	190	0.3981
0.4044	0.7323	200	0.3951
0.3989	0.7689	210	0.3912
0.3947	0.8055	220	0.3895
0.3945	0.8421	230	0.3868
0.3876	0.8787	240	0.3849
0.3877	0.9153	250	0.3839
0.3922	0.9519	260	0.3817
0.3844	0.9886	270	0.3796
0.3491	1.0252	280	0.3832
0.3291	1.0618	290	0.3821
0.3267	1.0984	300	0.3803

Framework versions

PEFT 0.13.2
Transformers 4.45.2
Pytorch 2.4.1+cu121
Datasets 3.0.1
Tokenizers 0.20.0

Library: peft
License: apache-2.0
Created: Oct 26, 2024
Updated: Oct 26, 2024