BERT_TS_WordPiece_phonetic_wikitext_0.5

psktoure/BERT_TS_WordPiece_phonetic_wikitext_0.5

BERT_TS_WordPiece_phonetic_wikitext_0.5 is a fill-mask model from psktoure. Use it when you need the model to fill a missing word. It is set up for transformers.

should probably proofread and complete it, then remove this comment. --

Downloads · 30 days

108

82% of all-time downloads

All-time downloads

131

Public

Parameters

110M

175 GB on disk

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors438 MB · 100%

At a glance

Task: Fill-Mask
Library: transformers
Model type: bert
Access: Public
Created: Dec 17, 2024
Updated: Dec 31, 2024
SHA: 7bcbee10

Try a prompt

Model card

From the Hugging Face model README

BERT_TS_WordPiece_phonetic_wikitext_0.5

This model is a fine-tuned version of on an unknown dataset. It achieves the following results on the evaluation set:

Loss: 0.7137

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0001
train_batch_size: 256
eval_batch_size: 256
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-06
lr_scheduler_type: linear
lr_scheduler_warmup_steps: 10000
training_steps: 400000
mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
5.9031	0.3019	2000	5.7764
4.7698	0.6039	4000	4.6379
3.506	0.9058	6000	3.4083
1.8562	1.2077	8000	1.7176
1.576	1.5097	10000	1.4738
1.4442	1.8116	12000	1.3477
1.353	2.1135	14000	1.2620
1.301	2.4155	16000	1.2197
1.2539	2.7174	18000	1.1761
1.2099	3.0193	20000	1.1443
1.178	3.3213	22000	1.1113
1.1546	3.6232	24000	1.0925
1.1365	3.9251	26000	1.0720
1.1139	4.2271	28000	1.0558
1.089	4.5290	30000	1.0308
1.0778	4.8309	32000	1.0352
1.0521	5.1329	34000	1.0068
1.0505	5.4348	36000	0.9877
1.0421	5.7367	38000	0.9729
1.0221	6.0386	40000	0.9750
1.0157	6.3406	42000	0.9692
1.0023	6.6425	44000	0.9494
0.9957	6.9444	46000	0.9545
0.9843	7.2464	48000	0.9375
0.9771	7.5483	50000	0.9366
0.976	7.8502	52000	0.9235
0.9664	8.1522	54000	0.9287
0.9563	8.4541	56000	0.9133
0.9524	8.7560	58000	0.9046
0.9325	9.0580	60000	0.9106
0.9385	9.3599	62000	0.8949
0.9335	9.6618	64000	0.8888
0.9322	9.9638	66000	0.8866
0.9171	10.2657	68000	0.8965
0.922	10.5676	70000	0.8890
0.9151	10.8696	72000	0.8757
0.9014	11.1715	74000	0.8681
0.9058	11.4734	76000	0.8703
0.8991	11.7754	78000	0.8732
0.891	12.0773	80000	0.8563
0.892	12.3792	82000	0.8544
0.8846	12.6812	84000	0.8663
0.8868	12.9831	86000	0.8547
0.8763	13.2850	88000	0.8534
0.8777	13.5870	90000	0.8543
0.8717	13.8889	92000	0.8456
0.8658	14.1908	94000	0.8402
0.8681	14.4928	96000	0.8403
0.8683	14.7947	98000	0.8353
0.856	15.0966	100000	0.8346
0.8572	15.3986	102000	0.8406
0.8616	15.7005	104000	0.8302
0.8564	16.0024	106000	0.8280
0.8459	16.3043	108000	0.8280
0.8453	16.6063	110000	0.8250
0.8477	16.9082	112000	0.8269
0.8373	17.2101	114000	0.8177
0.8416	17.5121	116000	0.8185
0.8367	17.8140	118000	0.8108
0.8273	18.1159	120000	0.8334
0.8302	18.4179	122000	0.8194
0.8274	18.7198	124000	0.8123
0.8176	19.0217	126000	0.8095
0.8173	19.3237	128000	0.8129
0.8194	19.6256	130000	0.8069
0.8203	19.9275	132000	0.8218
0.8115	20.2295	134000	0.7983
0.8135	20.5314	136000	0.7977
0.8111	20.8333	138000	0.8019
0.8031	21.1353	140000	0.8006
0.8031	21.4372	142000	0.8038
0.809	21.7391	144000	0.8068
0.7937	22.0411	146000	0.8109
0.7928	22.3430	148000	0.7942
0.7971	22.6449	150000	0.7979
0.7924	22.9469	152000	0.7930
0.791	23.2488	154000	0.7964
0.7897	23.5507	156000	0.8019
0.7907	23.8527	158000	0.7973
0.7765	24.1546	160000	0.7847
0.7831	24.4565	162000	0.7845
0.7865	24.7585	164000	0.7861
0.7738	25.0604	166000	0.7883
0.777	25.3623	168000	0.7938
0.7785	25.6643	170000	0.7742
0.7771	25.9662	172000	0.7877
0.7702	26.2681	174000	0.7904
0.7733	26.5700	176000	0.7780
0.7708	26.8720	178000	0.7762
0.765	27.1739	180000	0.7804
0.7645	27.4758	182000	0.7890
0.7629	27.7778	184000	0.7645
0.7556	28.0797	186000	0.7789
0.7571	28.3816	188000	0.7766
0.7573	28.6836	190000	0.7714
0.7599	28.9855	192000	0.7721
0.7525	29.2874	194000	0.7725
0.7538	29.5894	196000	0.7779
0.7547	29.8913	198000	0.7750
0.743	30.1932	200000	0.7640
0.7495	30.4952	202000	0.7626
0.7458	30.7971	204000	0.7760
0.7403	31.0990	206000	0.7731
0.7434	31.4010	208000	0.7669
0.7432	31.7029	210000	0.7607
0.7396	32.0048	212000	0.7621
0.743	32.3068	214000	0.7639
0.7406	32.6087	216000	0.7608
0.741	32.9106	218000	0.7512
0.7317	33.2126	220000	0.7621
0.731	33.5145	222000	0.7618
0.7359	33.8164	224000	0.7635
0.7229	34.1184	226000	0.7656
0.7273	34.4203	228000	0.7470
0.7347	34.7222	230000	0.7596
0.7219	35.0242	232000	0.7649
0.7233	35.3261	234000	0.7614
0.7239	35.6280	236000	0.7470
0.7241	35.9300	238000	0.7489
0.7184	36.2319	240000	0.7516
0.7189	36.5338	242000	0.7628
0.7217	36.8357	244000	0.7515
0.7146	37.1377	246000	0.7520
0.7168	37.4396	248000	0.7551
0.7155	37.7415	250000	0.7469
0.7098	38.0435	252000	0.7573
0.7105	38.3454	254000	0.7441
0.7078	38.6473	256000	0.7547
0.7143	38.9493	258000	0.7537
0.7033	39.2512	260000	0.7570
0.7083	39.5531	262000	0.7564
0.7062	39.8551	264000	0.7515
0.701	40.1570	266000	0.7429
0.7016	40.4589	268000	0.7518
0.7056	40.7609	270000	0.7469
0.701	41.0628	272000	0.7468
0.6993	41.3647	274000	0.7434
0.7055	41.6667	276000	0.7405
0.7024	41.9686	278000	0.7410
0.6972	42.2705	280000	0.7408
0.6965	42.5725	282000	0.7459
0.6963	42.8744	284000	0.7444
0.689	43.1763	286000	0.7415
0.6887	43.4783	288000	0.7279
0.6893	43.7802	290000	0.7395
0.689	44.0821	292000	0.7460
0.6879	44.3841	294000	0.7430
0.6906	44.6860	296000	0.7387
0.6919	44.9879	298000	0.7414
0.6818	45.2899	300000	0.7318
0.6913	45.5918	302000	0.7304
0.6863	45.8937	304000	0.7325
0.6794	46.1957	306000	0.7338
0.6792	46.4976	308000	0.7482
0.6798	46.7995	310000	0.7425
0.6805	47.1014	312000	0.7299
0.6811	47.4034	314000	0.7341
0.6813	47.7053	316000	0.7335
0.6746	48.0072	318000	0.7370
0.6782	48.3092	320000	0.7292
0.6783	48.6111	322000	0.7358
0.6767	48.9130	324000	0.7353
0.6713	49.2150	326000	0.7330
0.6709	49.5169	328000	0.7334
0.6721	49.8188	330000	0.7323
0.6666	50.1208	332000	0.7195
0.6727	50.4227	334000	0.7216
0.6704	50.7246	336000	0.7297
0.6677	51.0266	338000	0.7274
0.6669	51.3285	340000	0.7211
0.6724	51.6304	342000	0.7320
0.6678	51.9324	344000	0.7253
0.6618	52.2343	346000	0.7210
0.6632	52.5362	348000	0.7252
0.662	52.8382	350000	0.7344
0.6635	53.1401	352000	0.7254
0.6595	53.4420	354000	0.7232
0.6636	53.7440	356000	0.7250
0.6573	54.0459	358000	0.7179
0.6571	54.3478	360000	0.7199
0.6572	54.6498	362000	0.7236
0.6597	54.9517	364000	0.7231
0.6568	55.2536	366000	0.7199
0.6566	55.5556	368000	0.7205
0.6571	55.8575	370000	0.7234
0.6593	56.1594	372000	0.7166
0.6531	56.4614	374000	0.7157
0.6571	56.7633	376000	0.7212
0.6542	57.0652	378000	0.7194
0.6554	57.3671	380000	0.7124
0.6507	57.6691	382000	0.7182
0.6531	57.9710	384000	0.7216
0.6533	58.2729	386000	0.7087
0.6507	58.5749	388000	0.7152
0.6478	58.8768	390000	0.7198
0.6486	59.1787	392000	0.7228
0.6512	59.4807	394000	0.7137
0.6478	59.7826	396000	0.7221
0.6459	60.0845	398000	0.7147
0.6468	60.3865	400000	0.7137

Framework versions

Transformers 4.45.1
Pytorch 2.4.1+cu121
Datasets 3.0.1
Tokenizers 0.20.0

Task: Fill-Mask
Library: transformers
Type: bert
Created: Dec 17, 2024
Updated: Dec 31, 2024