zephyr-infoNCA-reward

ChenDRAG/zephyr-infoNCA-reward

zephyr-infoNCA-reward is a text generation model from ChenDRAG. Use it when you need the model to write or continue text. It is set up for transformers.

This model is a fine-tuned version of HuggingFaceH4/mistral-7b-sft-beta on the openbmb/UltraFeedback dataset. - Loss: 0.8810 - Loss/mini Gap Loss: 0.8810 - Loss/ori Loss: 1.1137 - Loss/reward Entrophy: 0.2326

Downloads · 30 days

6% of all-time downloads

All-time downloads

Public

Repo size

168 MB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors168 MB · 97%

At a glance

Task: Text Generation
Library: transformers
Model type: mistral
Access: Public
Created: Feb 8, 2024
Updated: Feb 8, 2024
SHA: f2db39b7

Try a prompt

Model card

From the Hugging Face model README

zephyr-infoNCA-reward

This model is a fine-tuned version of HuggingFaceH4/mistral-7b-sft-beta on the openbmb/UltraFeedback dataset.

Loss: 0.8810
Loss/mini Gap Loss: 0.8810
Loss/ori Loss: 1.1137
Loss/reward Entrophy: 0.2326
Regularization/forward Kl: 1.5849
Regularization/reverse Kl: 0.9146
Regularization/policy Data Loss: 3.2706
Regularization/reference Data Loss: 1.2660
Regularization/policy Ref Data Loss Gap: 2.0046
Mask/mask Ratio: 0.4577
Reward/reward A0: -0.9007
Reward/reward A1: -1.2463
Reward/reward A2: -1.5959
Reward/reward A3: -2.0882
Rewards/chosen: -0.9007
Rewards/rejected: -1.6434
Rewards/margins: 0.7428
Reward/a01 Acc: 0.6366
Reward/a02 Acc: 0.7334
Reward/a03 Acc: 0.8302
Rewards/accuracies: 0.7334

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-06
train_batch_size: 1
eval_batch_size: 1
seed: 42
distributed_type: multi-GPU
num_devices: 4
gradient_accumulation_steps: 8
total_train_batch_size: 32
total_eval_batch_size: 4
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.1
num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Loss/mini Gap Loss	Loss/ori Loss	Loss/reward Entrophy	Regularization/forward Kl	Regularization/reverse Kl	Regularization/policy Data Loss	Regularization/reference Data Loss	Regularization/policy Ref Data Loss Gap	Mask/mask Ratio	Reward/reward A0	Reward/reward A1	Reward/reward A2	Reward/reward A3	Rewards/chosen	Rewards/rejected	Rewards/margins	Reward/a01 Acc	Reward/a02 Acc	Reward/a03 Acc	Rewards/accuracies
1.1592	0.05	100	1.1483	1.1484	1.3811	0.2326	0.0008	0.0008	1.2693	1.2660	0.0033	0.4577	0.0031	-0.0005	-0.0032	-0.0066	0.0031	-0.0034	0.0065	0.5864	0.6667	0.7205	0.6579
1.0838	0.11	200	1.0772	1.0773	1.3100	0.2326	0.1510	0.1265	1.4842	1.2660	0.2182	0.4577	-0.1490	-0.2198	-0.2639	-0.3185	-0.1490	-0.2674	0.1184	0.6040	0.6698	0.7081	0.6606
1.0427	0.16	300	1.0091	1.0092	1.2419	0.2326	0.5873	0.4077	1.8854	1.2660	0.6194	0.4577	-0.4752	-0.6617	-0.7889	-0.9494	-0.4752	-0.8000	0.3248	0.6196	0.6744	0.7360	0.6767
0.9666	0.21	400	0.9712	0.9713	1.2039	0.2326	0.7687	0.4464	2.2361	1.2660	0.9701	0.4577	-0.5326	-0.7468	-0.9238	-1.1650	-0.5326	-0.9452	0.4126	0.6289	0.7013	0.7676	0.6993
0.984	0.27	500	0.9523	0.9524	1.1850	0.2326	0.8699	0.4759	2.4013	1.2660	1.1353	0.4577	-0.5793	-0.8081	-1.0134	-1.2919	-0.5793	-1.0378	0.4585	0.6242	0.7034	0.7831	0.7036
1.0017	0.32	600	0.9367	0.9368	1.1694	0.2326	1.0544	0.6109	2.6903	1.2660	1.4243	0.4577	-0.7541	-1.0241	-1.2660	-1.5769	-0.7541	-1.2890	0.5350	0.6413	0.7091	0.7836	0.7113
0.9615	0.37	700	0.9338	0.9338	1.1665	0.2326	1.2767	0.7017	3.0578	1.2660	1.7918	0.4577	-0.9159	-1.2048	-1.4643	-1.7939	-0.9159	-1.4877	0.5717	0.6289	0.7133	0.7867	0.7096
0.9292	0.42	800	0.9237	0.9237	1.1564	0.2326	1.3185	0.7646	3.1225	1.2660	1.8565	0.4577	-0.8569	-1.1333	-1.4081	-1.7547	-0.8569	-1.4320	0.5751	0.6284	0.7169	0.8043	0.7165
0.9366	0.48	900	0.9099	0.9100	1.1426	0.2326	1.3334	0.7449	2.9612	1.2660	1.6952	0.4577	-0.8158	-1.1198	-1.4146	-1.8111	-0.8158	-1.4485	0.6327	0.6387	0.7220	0.8121	0.7243
0.8746	0.53	1000	0.9005	0.9005	1.1332	0.2326	1.4735	0.8523	3.0808	1.2660	1.8148	0.4577	-0.8931	-1.2235	-1.5380	-1.9733	-0.8931	-1.5782	0.6852	0.6392	0.7319	0.8080	0.7264
0.8941	0.58	1100	0.8952	0.8952	1.1279	0.2326	1.4775	0.8426	3.1270	1.2660	1.8610	0.4577	-0.9341	-1.2736	-1.6024	-2.0415	-0.9341	-1.6392	0.7051	0.6413	0.7340	0.8111	0.7288
0.9201	0.64	1200	0.8891	0.8891	1.1218	0.2326	1.5023	0.8385	3.2583	1.2660	1.9923	0.4577	-0.9362	-1.2764	-1.6100	-2.0560	-0.9362	-1.6474	0.7112	0.6335	0.7329	0.8245	0.7303
0.8358	0.69	1300	0.8860	0.8861	1.1187	0.2326	1.6540	0.9301	3.2862	1.2660	2.0202	0.4577	-0.9350	-1.2850	-1.6319	-2.1211	-0.9350	-1.6793	0.7443	0.6423	0.7329	0.8214	0.7322
0.8829	0.74	1400	0.8846	0.8847	1.1174	0.2326	1.4174	0.8464	3.0760	1.2660	1.8100	0.4577	-0.8119	-1.1349	-1.4591	-1.9229	-0.8119	-1.5056	0.6938	0.6392	0.7381	0.8297	0.7357
0.8779	0.8	1500	0.8822	0.8823	1.1150	0.2326	1.6183	0.9325	3.3052	1.2660	2.0392	0.4577	-0.9158	-1.2611	-1.6110	-2.1030	-0.9158	-1.6583	0.7425	0.6387	0.7345	0.8261	0.7331
0.9388	0.85	1600	0.8818	0.8819	1.1145	0.2326	1.6409	0.9388	3.3318	1.2660	2.0658	0.4577	-0.9332	-1.2823	-1.6359	-2.1322	-0.9332	-1.6834	0.7502	0.6361	0.7319	0.8271	0.7317
0.8319	0.9	1700	0.8811	0.8812	1.1139	0.2326	1.5745	0.9076	3.2655	1.2660	1.9995	0.4577	-0.8984	-1.2427	-1.5909	-2.0806	-0.8984	-1.6380	0.7396	0.6356	0.7350	0.8307	0.7338
0.8719	0.96	1800	0.8809	0.8810	1.1137	0.2326	1.5827	0.9136	3.2695	1.2660	2.0034	0.4577	-0.8998	-1.2451	-1.5947	-2.0870	-0.8998	-1.6423	0.7424	0.6372	0.7340	0.8307	0.7339

Framework versions

Transformers 4.35.0
Pytorch 2.1.0+cu118
Datasets 2.14.6
Tokenizers 0.14.1

Task: Text Generation
Library: transformers
Type: mistral
Created: Feb 8, 2024
Updated: Feb 8, 2024