pythia410m_wikitext_attn_codebook_model

taufeeque/pythia410m_wikitext_attn_codebook_model

pythia410m_wikitext_attn_codebook_model is a machine learning model from taufeeque. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.

should probably proofread and complete it, then remove this comment. --

Downloads · 30 days

19% of all-time downloads

All-time downloads

Public

Repo size

2.7 GB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.bin2.7 GB · 100%

At a glance

Library: transformers
License: apache-2.0
Model type: codebook
Access: Public
Created: Nov 15, 2023
Updated: Nov 15, 2023
SHA: 0b846e55

Datasets

wikitext

Model card

From the Hugging Face model README

output_main

This model is a fine-tuned version of EleutherAI/pythia-410m-deduped on the wikitext wikitext-103-v1 dataset. It achieves the following results on the evaluation set:

Loss: 2.7446
Accuracy: 0.4668
Multicode K: 8
Dead Code Fraction/layer0: 1.0
Mse/layer0: 0.0
Input Norm/layer0: 0.0
Output Norm/layer0: 0.0
Dead Code Fraction/layer1: 1.0
Mse/layer1: 0.0
Input Norm/layer1: 0.0
Output Norm/layer1: 0.0
Dead Code Fraction/layer2: 1.0
Mse/layer2: 0.0
Input Norm/layer2: 0.0
Output Norm/layer2: 0.0
Dead Code Fraction/layer3: 1.0
Mse/layer3: 0.0
Input Norm/layer3: 0.0
Output Norm/layer3: 0.0
Dead Code Fraction/layer4: 1.0
Mse/layer4: 0.0
Input Norm/layer4: 0.0
Output Norm/layer4: 0.0
Dead Code Fraction/layer5: 1.0
Mse/layer5: 0.0
Input Norm/layer5: 0.0
Output Norm/layer5: 0.0
Dead Code Fraction/layer6: 1.0
Mse/layer6: 0.0
Input Norm/layer6: 0.0
Output Norm/layer6: 0.0
Dead Code Fraction/layer7: 1.0
Mse/layer7: 0.0
Input Norm/layer7: 0.0
Output Norm/layer7: 0.0
Dead Code Fraction/layer8: 1.0
Mse/layer8: 0.0
Input Norm/layer8: 0.0
Output Norm/layer8: 0.0
Dead Code Fraction/layer9: 1.0
Mse/layer9: 0.0
Input Norm/layer9: 0.0
Output Norm/layer9: 0.0
Dead Code Fraction/layer10: 1.0
Mse/layer10: 0.0
Input Norm/layer10: 0.0
Output Norm/layer10: 0.0
Dead Code Fraction/layer11: 1.0
Mse/layer11: 0.0
Input Norm/layer11: 0.0
Output Norm/layer11: 0.0
Dead Code Fraction/layer12: 1.0
Mse/layer12: 0.0
Input Norm/layer12: 0.0
Output Norm/layer12: 0.0
Dead Code Fraction/layer13: 1.0
Mse/layer13: 0.0
Input Norm/layer13: 0.0
Output Norm/layer13: 0.0
Dead Code Fraction/layer14: 1.0
Mse/layer14: 0.0
Input Norm/layer14: 0.0
Output Norm/layer14: 0.0
Dead Code Fraction/layer15: 1.0
Mse/layer15: 0.0
Input Norm/layer15: 0.0
Output Norm/layer15: 0.0
Dead Code Fraction/layer16: 1.0
Mse/layer16: 0.0
Input Norm/layer16: 0.0
Output Norm/layer16: 0.0
Dead Code Fraction/layer17: 1.0
Mse/layer17: 0.0
Input Norm/layer17: 0.0
Output Norm/layer17: 0.0
Dead Code Fraction/layer18: 1.0
Mse/layer18: 0.0
Input Norm/layer18: 0.0
Output Norm/layer18: 0.0
Dead Code Fraction/layer19: 1.0
Mse/layer19: 0.0
Input Norm/layer19: 0.0
Output Norm/layer19: 0.0
Dead Code Fraction/layer20: 1.0
Mse/layer20: 0.0
Input Norm/layer20: 0.0
Output Norm/layer20: 0.0
Dead Code Fraction/layer21: 1.0
Mse/layer21: 0.0
Input Norm/layer21: 0.0
Output Norm/layer21: 0.0
Dead Code Fraction/layer22: 1.0
Mse/layer22: 0.0
Input Norm/layer22: 0.0
Output Norm/layer22: 0.0
Dead Code Fraction/layer23: 1.0
Mse/layer23: 0.0
Input Norm/layer23: 0.0
Output Norm/layer23: 0.0

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0005
train_batch_size: 12
eval_batch_size: 8
seed: 42
distributed_type: multi-GPU
num_devices: 4
total_train_batch_size: 48
total_eval_batch_size: 32
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: constant
training_steps: 15000

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Multicode K	Dead Code Fraction/layer0	Dead Code Fraction/layer1	Dead Code Fraction/layer2	Dead Code Fraction/layer3	Dead Code Fraction/layer4	Dead Code Fraction/layer5	Dead Code Fraction/layer6	Dead Code Fraction/layer7	Dead Code Fraction/layer8	Dead Code Fraction/layer9	Dead Code Fraction/layer10	Dead Code Fraction/layer11	Dead Code Fraction/layer12	Dead Code Fraction/layer13	Dead Code Fraction/layer14	Dead Code Fraction/layer15	Dead Code Fraction/layer16	Dead Code Fraction/layer17	Dead Code Fraction/layer18	Dead Code Fraction/layer19	Dead Code Fraction/layer20	Dead Code Fraction/layer21	Dead Code Fraction/layer22	Dead Code Fraction/layer23
7.6233	0.02	50	5.8594	0.1632	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
4.8681	0.04	100	3.9806	0.3218	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.8442	0.06	150	3.6302	0.3575	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.6253	0.08	200	3.4840	0.3722	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.5035	0.1	250	3.3861	0.3824	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.4446	0.13	300	3.3388	0.3875	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.3923	0.15	350	3.2929	0.3929	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.3488	0.17	400	3.2623	0.3978	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.3177	0.19	450	3.2322	0.3995	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.2894	0.21	500	3.2090	0.4022	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.2724	0.23	550	3.1913	0.4047	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.2613	0.25	600	3.1640	0.4081	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.2317	0.27	650	3.1502	0.4097	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.2188	0.29	700	3.1365	0.4117	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.2161	0.31	750	3.1243	0.4147	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.1922	0.33	800	3.1104	0.4158	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.1722	0.35	850	3.0969	0.4158	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.1726	0.38	900	3.0818	0.4187	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.1672	0.4	950	3.0697	0.4197	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.1499	0.42	1000	3.0661	0.4205	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.137	0.44	1050	3.0529	0.4216	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.1325	0.46	1100	3.0534	0.4227	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.1134	0.48	1150	3.0337	0.4245	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.0942	0.5	1200	3.0296	0.4252	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.1078	0.52	1250	3.0219	0.4257	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.086	0.54	1300	3.0091	0.4278	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.0909	0.56	1350	3.0010	0.4289	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.0855	0.58	1400	2.9934	0.4288	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.065	0.61	1450	2.9869	0.4306	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.0722	0.63	1500	2.9838	0.4306	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.0536	0.65	1550	2.9825	0.4301	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.0505	0.67	1600	2.9796	0.4307	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.0475	0.69	1650	2.9699	0.4328	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.0322	0.71	1700	2.9611	0.4334	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.0316	0.73	1750	2.9501	0.4339	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.02	0.75	1800	2.9468	0.4347	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.0245	0.77	1850	2.9405	0.4355	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.0247	0.79	1900	2.9340	0.4362	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.0201	0.81	1950	2.9285	0.4371	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.0107	0.84	2000	2.9189	0.4382	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
3.003	0.86	2050	2.9210	0.4388	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
2.9918	0.88	2100	2.9100	0.4389	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
2.9853	0.9	2150	2.9018	0.4399	8	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0

The model card is truncated. Read the rest on Hugging Face.

Library: transformers
Type: codebook
License: apache-2.0
Created: Nov 15, 2023
Updated: Nov 15, 2023