pruned-mt5-small

X-Wang/pruned-mt5-small

pruned-mt5-small is a machine learning model from X-Wang. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.

should probably proofread and complete it, then remove this comment. --

Downloads · 30 days

2% of all-time downloads

All-time downloads

785

Public

Repo size

57.9 GB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.bin342 MB · 99%

At a glance

Library: transformers
Model type: mt5
Access: Public
Created: Jul 8, 2023
Updated: Jul 28, 2023
SHA: 89b7d23f

Datasets

Helsinki-NLP/tatoeba_mt

Model card

From the Hugging Face model README

pruned-mt5-small

This model is a fine-tuned version of X-Wang/pruned-mt5-small on the None dataset. It achieves the following results on the evaluation set:

Loss: 2.4431
Bleu: 11.4084
Gen Len: 16.1053

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0005
train_batch_size: 12
eval_batch_size: 12
seed: 42
gradient_accumulation_steps: 2
total_train_batch_size: 24
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.01
num_epochs: 2

Training results

Training Loss	Epoch	Step	Validation Loss	Bleu	Gen Len
3.3446	0.07	2000	2.9103	10.3957	16.0567
2.8425	0.14	4000	2.8570	10.5695	16.1895
3.186	0.21	6000	2.8137	10.5958	16.1523
2.788	0.28	8000	2.7593	10.7553	16.0138
2.9075	0.35	10000	2.7266	10.9199	16.2016
3.0579	0.42	12000	2.7030	10.6	16.0496
2.3618	0.49	14000	2.6547	10.8026	16.0412
3.079	0.56	16000	2.6441	10.7945	16.1148
2.7597	0.63	18000	2.6244	10.5877	16.0507
2.8533	0.7	20000	2.6049	10.9986	16.1145
2.843	0.77	22000	2.5836	10.9173	16.0826
2.8268	0.84	24000	2.5685	10.8136	16.0516
2.7021	0.91	26000	2.5509	11.326	16.0554
3.338	0.98	28000	2.5289	11.1485	16.0333
2.7374	1.05	30000	2.5220	11.0166	16.0998
2.7996	1.12	32000	2.5077	11.1316	16.131
2.6897	1.19	34000	2.4994	11.0811	16.1139
2.4107	1.26	36000	2.4877	11.2641	16.142
2.7695	1.33	38000	2.4756	11.2135	16.0977
3.3271	1.41	40000	2.4658	11.3328	16.0953
2.2641	1.48	42000	2.4612	11.3065	16.0549
2.6594	1.55	44000	2.4556	11.2684	16.1371
2.7322	1.62	46000	2.4520	11.3739	16.1058
2.6824	1.69	48000	2.4462	11.3335	16.1043
2.3369	1.76	50000	2.4455	11.3851	16.1239
2.9537	1.83	52000	2.4430	11.4026	16.0858
2.3928	1.9	54000	2.4433	11.301	16.1129
2.4714	1.97	56000	2.4431	11.4084	16.1053

Framework versions

Transformers 4.31.0
Pytorch 2.0.0
Datasets 2.13.1
Tokenizers 0.13.3

Library: transformers
Type: mt5
Languages: ja, zh
Created: Jul 8, 2023
Updated: Jul 28, 2023