BERT_BPE_phonetic_bookcorpus_0.1

psktoure/BERT_BPE_phonetic_bookcorpus_0.1

BERT_BPE_phonetic_bookcorpus_0.1 is a fill-mask model from psktoure. Use it when you need the model to fill a missing word. It is set up for transformers.

should probably proofread and complete it, then remove this comment. --

Downloads · 30 days

30% of all-time downloads

All-time downloads

Public

Parameters

110M

85.9 GB on disk

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors438 MB · 100%

At a glance

Task: Fill-Mask
Library: transformers
Model type: bert
Access: Public
Created: Nov 27, 2024
Updated: Dec 3, 2024
SHA: 84b762fa

Try a prompt

Model card

From the Hugging Face model README

BERT_WordLevel_phonetic_bookcorpus_0.1

This model is a fine-tuned version of on the None dataset. It achieves the following results on the evaluation set:

Loss: 0.7433

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0001
train_batch_size: 320
eval_batch_size: 320
seed: 42
gradient_accumulation_steps: 4
total_train_batch_size: 1280
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-06
lr_scheduler_type: linear
lr_scheduler_warmup_steps: 10000
num_epochs: 40
mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
4.9362	0.2254	2000	4.8888
2.8229	0.4507	4000	2.6678
2.0553	0.6761	6000	1.9699
1.787	0.9015	8000	1.7132
1.6365	1.1269	10000	1.5669
1.5186	1.3522	12000	1.4545
1.4419	1.5776	14000	1.3796
1.3888	1.8030	16000	1.3205
1.3361	2.0283	18000	1.2823
1.3003	2.2537	20000	1.2412
1.2662	2.4791	22000	1.2131
1.2457	2.7045	24000	1.1911
1.221	2.9298	26000	1.1588
1.1956	3.1552	28000	1.1438
1.1782	3.3806	30000	1.1297
1.1686	3.6059	32000	1.1093
1.1558	3.8313	34000	1.1021
1.1405	4.0567	36000	1.0924
1.1303	4.2821	38000	1.0750
1.1171	4.5074	40000	1.0668
1.1016	4.7328	42000	1.0545
1.0946	4.9582	44000	1.0437
1.0868	5.1835	46000	1.0358
1.0743	5.4089	48000	1.0317
1.0695	5.6343	50000	1.0218
1.0626	5.8597	52000	1.0163
1.0533	6.0850	54000	1.0065
1.0493	6.3104	56000	0.9993
1.0436	6.5358	58000	0.9905
1.0351	6.7611	60000	0.9859
1.0285	6.9865	62000	0.9855
1.0194	7.2119	64000	0.9787
1.0197	7.4372	66000	0.9705
1.0166	7.6626	68000	0.9664
1.0088	7.8880	70000	0.9609
1.0026	8.1134	72000	0.9563
0.9955	8.3387	74000	0.9553
0.9921	8.5641	76000	0.9482
0.9917	8.7895	78000	0.9453
0.9782	9.0148	80000	0.9423
0.9829	9.2402	82000	0.9345
0.9722	9.4656	84000	0.9334
0.9784	9.6910	86000	0.9242
0.971	9.9163	88000	0.9212
0.9631	10.1417	90000	0.9238
0.9619	10.3671	92000	0.9147
0.9564	10.5924	94000	0.9144
0.9503	10.8178	96000	0.9107
0.9453	11.0432	98000	0.9050
0.9461	11.2686	100000	0.9048
0.9409	11.4939	102000	0.9038
0.9475	11.7193	104000	0.8949
0.9396	11.9447	106000	0.8964
0.9289	12.1700	108000	0.8943
0.9323	12.3954	110000	0.8872
0.9298	12.6208	112000	0.8885
0.9256	12.8462	114000	0.8835
0.9228	13.0715	116000	0.8833
0.9174	13.2969	118000	0.8788
0.9193	13.5223	120000	0.8802
0.9135	13.7476	122000	0.8772
0.9164	13.9730	124000	0.8754
0.91	14.1984	126000	0.8648
0.9062	14.4238	128000	0.8700
0.905	14.6491	130000	0.8635
0.9	14.8745	132000	0.8602
0.9014	15.0999	134000	0.8643
0.8981	15.3252	136000	0.8591
0.8996	15.5506	138000	0.8590
0.8985	15.7760	140000	0.8561
0.8914	16.0014	142000	0.8570
0.8922	16.2267	144000	0.8496
0.8927	16.4521	146000	0.8500
0.8884	16.6775	148000	0.8467
0.887	16.9028	150000	0.8433
0.8795	17.1282	152000	0.8374
0.8813	17.3536	154000	0.8408
0.8789	17.5790	156000	0.8384
0.8772	17.8043	158000	0.8380
0.8801	18.0297	160000	0.8390
0.8745	18.2551	162000	0.8360
0.8714	18.4804	164000	0.8330
0.8698	18.7058	166000	0.8288
0.8665	18.9312	168000	0.8317
0.8688	19.1565	170000	0.8274
0.8647	19.3819	172000	0.8231
0.8607	19.6073	174000	0.8275
0.8641	19.8327	176000	0.8222
0.8599	20.0580	178000	0.8245
0.8568	20.2834	180000	0.8182
0.8557	20.5088	182000	0.8192
0.856	20.7341	184000	0.8152
0.856	20.9595	186000	0.8222
0.8498	21.1849	188000	0.8121
0.8529	21.4103	190000	0.8116
0.8489	21.6356	192000	0.8128
0.8483	21.8610	194000	0.8075
0.8451	22.0864	196000	0.8082
0.845	22.3117	198000	0.8098
0.843	22.5371	200000	0.8080
0.8371	22.7625	202000	0.8041
0.8373	22.9879	204000	0.8018
0.8333	23.2132	206000	0.8020
0.8359	23.4386	208000	0.8006
0.834	23.6640	210000	0.7972
0.8347	23.8893	212000	0.7966
0.8326	24.1147	214000	0.7966
0.828	24.3401	216000	0.7938
0.8315	24.5655	218000	0.7941
0.8307	24.7908	220000	0.7952
0.8223	25.0162	222000	0.7894
0.8265	25.2416	224000	0.7901
0.8243	25.4669	226000	0.7905
0.8258	25.6923	228000	0.7898
0.8227	25.9177	230000	0.7864
0.8225	26.1431	232000	0.7894
0.8228	26.3684	234000	0.7867
0.8202	26.5938	236000	0.7822
0.8152	26.8192	238000	0.7851
0.8137	27.0445	240000	0.7845
0.8164	27.2699	242000	0.7806
0.8152	27.4953	244000	0.7806
0.813	27.7207	246000	0.7816
0.8105	27.9460	248000	0.7828
0.8098	28.1714	250000	0.7810
0.8052	28.3968	252000	0.7749
0.8063	28.6221	254000	0.7746
0.8122	28.8475	256000	0.7726
0.8089	29.0729	258000	0.7720
0.8058	29.2983	260000	0.7742
0.808	29.5236	262000	0.7720
0.8066	29.7490	264000	0.7691
0.805	29.9744	266000	0.7728
0.8005	30.1997	268000	0.7660
0.8	30.4251	270000	0.7695
0.8046	30.6505	272000	0.7692
0.7989	30.8758	274000	0.7665
0.7995	31.1012	276000	0.7654
0.7989	31.3266	278000	0.7606
0.7938	31.5520	280000	0.7618
0.7952	31.7773	282000	0.7679
0.7926	32.0027	284000	0.7606
0.7903	32.2281	286000	0.7600
0.7906	32.4534	288000	0.7568
0.7922	32.6788	290000	0.7575
0.7909	32.9042	292000	0.7572
0.7873	33.1296	294000	0.7552
0.7904	33.3549	296000	0.7548
0.7884	33.5803	298000	0.7573
0.7863	33.8057	300000	0.7564
0.7842	34.0310	302000	0.7528
0.7864	34.2564	304000	0.7532
0.7869	34.4818	306000	0.7514
0.7835	34.7072	308000	0.7506
0.7829	34.9325	310000	0.7532
0.7792	35.1579	312000	0.7532
0.7831	35.3833	314000	0.7516
0.7812	35.6086	316000	0.7527
0.7766	35.8340	318000	0.7474
0.7787	36.0594	320000	0.7482
0.779	36.2848	322000	0.7504
0.78	36.5101	324000	0.7420
0.7733	36.7355	326000	0.7462
0.78	36.9609	328000	0.7466
0.7762	37.1862	330000	0.7463
0.7734	37.4116	332000	0.7471
0.7766	37.6370	334000	0.7477
0.7728	37.8624	336000	0.7468
0.7751	38.0877	338000	0.7450
0.7739	38.3131	340000	0.7457
0.7733	38.5385	342000	0.7452
0.7734	38.7638	344000	0.7422
0.7748	38.9892	346000	0.7433
0.7737	39.2146	348000	0.7441
0.7744	39.4400	350000	0.7418
0.7736	39.6653	352000	0.7404
0.7762	39.8907	354000	0.7433

Framework versions

Transformers 4.45.1
Pytorch 2.4.1+cu121
Datasets 3.0.1
Tokenizers 0.20.0

Task: Fill-Mask
Library: transformers
Type: bert
Created: Nov 27, 2024
Updated: Dec 3, 2024