hbertv1-massive-intermediate_KD_new_2

gokuls/hbertv1-massive-intermediate_KD_new_2

hbertv1-massive-intermediate_KD_new_2 is a text classification model from gokuls. Use it when you need a label for a piece of text. It is set up for transformers.

should probably proofread and complete it, then remove this comment. --

Downloads · 30 days

18% of all-time downloads

All-time downloads

Public

Parameters

104M

10.4 GB on disk

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors417 MB · 100%

At a glance

Task: Text Classification
Library: transformers
Model type: hybridbert
Access: Public
Created: Nov 26, 2023
Updated: Nov 26, 2023
SHA: 33fd54e1

Try a prompt

Base models

gokuls/HBERTv1_48_L10_H768_A12

Datasets

massive

Model card

From the Hugging Face model README

hbertv1-massive-intermediate_KD_new_2

This model is a fine-tuned version of gokuls/HBERTv1_48_L10_H768_A12 on the massive dataset. It achieves the following results on the evaluation set:

Loss: 1.3102
Accuracy: 0.8342

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-05
train_batch_size: 64
eval_batch_size: 64
seed: 33
distributed_type: multi-GPU
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 50

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
4.5836	1.0	180	3.4660	0.2710
3.38	2.0	360	2.7802	0.4324
2.7571	3.0	540	2.3906	0.5991
2.3743	4.0	720	2.1148	0.7029
2.1481	5.0	900	2.0007	0.7245
1.9762	6.0	1080	1.9660	0.7467
1.8702	7.0	1260	1.8680	0.7619
1.759	8.0	1440	1.8192	0.7806
1.6949	9.0	1620	1.7677	0.7949
1.6253	10.0	1800	1.7452	0.7885
1.5849	11.0	1980	1.7075	0.8023
1.5239	12.0	2160	1.6915	0.7939
1.4768	13.0	2340	1.6821	0.8067
1.4474	14.0	2520	1.7201	0.7944
1.424	15.0	2700	1.6538	0.8096
1.3839	16.0	2880	1.5979	0.8141
1.3537	17.0	3060	1.6254	0.8062
1.3422	18.0	3240	1.6386	0.8077
1.3166	19.0	3420	1.6048	0.8141
1.2923	20.0	3600	1.5927	0.8146
1.2722	21.0	3780	1.5544	0.8180
1.2513	22.0	3960	1.5904	0.8077
1.2286	23.0	4140	1.5506	0.8195
1.2056	24.0	4320	1.5547	0.8146
1.1941	25.0	4500	1.5258	0.8224
1.1701	26.0	4680	1.4975	0.8224
1.1582	27.0	4860	1.4945	0.8200
1.1367	28.0	5040	1.4888	0.8219
1.127	29.0	5220	1.4596	0.8254
1.1126	30.0	5400	1.4686	0.8175
1.0922	31.0	5580	1.4934	0.8200
1.0809	32.0	5760	1.4370	0.8249
1.0715	33.0	5940	1.4305	0.8234
1.0572	34.0	6120	1.4255	0.8273
1.0429	35.0	6300	1.4042	0.8249
1.0375	36.0	6480	1.4004	0.8190
1.0242	37.0	6660	1.3849	0.8269
1.0132	38.0	6840	1.3777	0.8288
1.0085	39.0	7020	1.3731	0.8273
0.9964	40.0	7200	1.3647	0.8278
0.9867	41.0	7380	1.3655	0.8239
0.9787	42.0	7560	1.3542	0.8293
0.9692	43.0	7740	1.3449	0.8278
0.9646	44.0	7920	1.3402	0.8283
0.959	45.0	8100	1.3360	0.8288
0.9482	46.0	8280	1.3289	0.8303
0.9503	47.0	8460	1.3173	0.8328
0.9428	48.0	8640	1.3152	0.8333
0.9416	49.0	8820	1.3102	0.8342
0.9348	50.0	9000	1.3133	0.8328

Framework versions

Transformers 4.35.2
Pytorch 1.14.0a0+410ce96
Datasets 2.15.0
Tokenizers 0.15.0

Task: Text Classification
Library: transformers
Type: hybridbert
Created: Nov 26, 2023
Updated: Nov 26, 2023