zephyr-infoNCA-preference

ChenDRAG/zephyr-infoNCA-preference

zephyr-infoNCA-preference is a text generation model from ChenDRAG. Use it when you need the model to write or continue text. It is set up for transformers.

This model is a fine-tuned version of HuggingFaceH4/mistral-7b-sft-beta on the openbmb/UltraFeedback dataset. It achieves the following results on the evaluation set: - Loss: 0.4575 - Rewards/chosen: -0.8931

Downloads · 30 days

5% of all-time downloads

All-time downloads

282

Public

Repo size

168 MB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors168 MB · 97%

At a glance

Task: Text Generation
Library: transformers
Model type: mistral
Access: Public
Created: Feb 8, 2024
Updated: Feb 8, 2024
SHA: d5a8d91d

Try a prompt

Model card

From the Hugging Face model README

zephyr-infoNCA-preference

This model is a fine-tuned version of HuggingFaceH4/mistral-7b-sft-beta on the openbmb/UltraFeedback dataset. It achieves the following results on the evaluation set:

Loss: 0.4575
Rewards/chosen: -0.8931
Rewards/rejected: -2.0138
Rewards/accuracies: 0.7745
Rewards/margins: 1.1206
Verify/constant 1: 1.0
Verify/constant 1len: 1000.0
Logps/rejected: -434.5525
Logps/chosen: -364.4662
Verify/bz: 1.0
Verify/gather Bz: 2.0
Regularization/forward Kl: 2.0564
Regularization/reverse Kl: 1.0252
Regularization/policy Data Loss: 3.8558
Regularization/reference Data Loss: 1.3337
Regularization/policy Ref Data Loss Gap: 2.5221
Mask/mask Ratio: 0.4809

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-06
train_batch_size: 1
eval_batch_size: 1
seed: 42
distributed_type: multi-GPU
num_devices: 2
gradient_accumulation_steps: 16
total_train_batch_size: 32
total_eval_batch_size: 2
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.1
num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Verify/constant 1	Verify/constant 1len	Logps/rejected	Logps/chosen	Verify/bz	Verify/gather Bz	Regularization/forward Kl	Regularization/reverse Kl	Regularization/policy Data Loss	Regularization/reference Data Loss	Regularization/policy Ref Data Loss Gap	Mask/mask Ratio
0.6892	0.05	100	0.6881	0.0039	-0.0063	0.7145	0.0102	1.0	1000.0	-233.8040	-274.7605	1.0	2.0	0.0009	0.0009	1.3405	1.3337	0.0068	0.4809
0.6259	0.1	200	0.6258	-0.1279	-0.2905	0.7145	0.1627	1.0	1000.0	-262.2266	-287.9373	1.0	2.0	0.1727	0.1289	1.6331	1.3337	0.2994	0.4809
0.5436	0.15	300	0.5495	-0.4736	-0.9395	0.7415	0.4659	1.0	1000.0	-327.1224	-322.5125	1.0	2.0	0.6904	0.3995	2.2940	1.3337	0.9603	0.4809
0.5492	0.21	400	0.5161	-0.5783	-1.2015	0.7545	0.6232	1.0	1000.0	-353.3223	-332.9807	1.0	2.0	0.9794	0.5146	2.7574	1.3337	1.4237	0.4809
0.521	0.26	500	0.4982	-0.7257	-1.5000	0.7595	0.7743	1.0	1000.0	-383.1716	-347.7220	1.0	2.0	1.2016	0.5622	3.0006	1.3337	1.6669	0.4809
0.5152	0.31	600	0.4887	-0.6594	-1.4497	0.7685	0.7903	1.0	1000.0	-378.1454	-341.0961	1.0	2.0	1.2196	0.6044	3.0235	1.3337	1.6897	0.4809
0.4862	0.36	700	0.4857	-0.7064	-1.5442	0.7655	0.8378	1.0	1000.0	-387.5948	-345.7939	1.0	2.0	1.2568	0.6231	3.2214	1.3337	1.8877	0.4809
0.4632	0.41	800	0.4803	-0.6298	-1.4654	0.7755	0.8356	1.0	1000.0	-379.7145	-338.1303	1.0	2.0	1.3128	0.7041	2.8330	1.3337	1.4993	0.4809
0.4912	0.46	900	0.4707	-0.7165	-1.6486	0.7750	0.9321	1.0	1000.0	-398.0345	-346.8000	1.0	2.0	1.4120	0.7160	3.0682	1.3337	1.7345	0.4809
0.4588	0.52	1000	0.4680	-0.8531	-1.8542	0.7690	1.0011	1.0	1000.0	-418.5936	-360.4624	1.0	2.0	1.6382	0.8346	3.5448	1.3337	2.2111	0.4809
0.4956	0.57	1100	0.4650	-0.7990	-1.7772	0.7790	0.9781	1.0	1000.0	-410.8913	-355.0567	1.0	2.0	1.6270	0.8004	3.5035	1.3337	2.1698	0.4809
0.4738	0.62	1200	0.4629	-0.8068	-1.8169	0.7705	1.0102	1.0	1000.0	-414.8670	-355.8280	1.0	2.0	1.7938	0.8907	3.6708	1.3337	2.3371	0.4809
0.4657	0.67	1300	0.4622	-0.8659	-1.9282	0.7655	1.0623	1.0	1000.0	-425.9926	-361.7412	1.0	2.0	1.9375	0.9455	3.7639	1.3337	2.4301	0.4809
0.4938	0.72	1400	0.4586	-0.8258	-1.9093	0.7745	1.0834	1.0	1000.0	-424.0995	-357.7357	1.0	2.0	1.8620	0.9612	3.5611	1.3337	2.2274	0.4809
0.4511	0.77	1500	0.4580	-0.8174	-1.8815	0.7765	1.0641	1.0	1000.0	-421.3289	-356.8928	1.0	2.0	1.8762	0.9513	3.6341	1.3337	2.3003	0.4809
0.4724	0.83	1600	0.4573	-0.8790	-1.9952	0.7735	1.1162	1.0	1000.0	-432.6913	-363.0503	1.0	2.0	2.0060	1.0139	3.7650	1.3337	2.4312	0.4809
0.5045	0.88	1700	0.4572	-0.8903	-2.0141	0.7725	1.1238	1.0	1000.0	-434.5795	-364.1794	1.0	2.0	2.0502	1.0267	3.8128	1.3337	2.4790	0.4809
0.5007	0.93	1800	0.4577	-0.9008	-2.0247	0.7715	1.1239	1.0	1000.0	-435.6480	-365.2350	1.0	2.0	2.0707	1.0309	3.8706	1.3337	2.5369	0.4809
0.4747	0.98	1900	0.4576	-0.8929	-2.0129	0.7735	1.1200	1.0	1000.0	-434.4668	-364.4426	1.0	2.0	2.0555	1.0247	3.8552	1.3337	2.5215	0.4809

Framework versions

Transformers 4.35.0
Pytorch 2.1.0
Datasets 2.14.6
Tokenizers 0.14.1

Task: Text Generation
Library: transformers
Type: mistral
Created: Feb 8, 2024
Updated: Feb 8, 2024