Downloads · 30 days
13
2% of all-time downloads
electra_small_discriminator
junnyu/electra_small_discriminator
electra_small_discriminator is a machine learning model from junnyu. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.
- 数据集 openwebtext - 训练batchsize 256 - 学习率lr 5e-4 - 最大句子长度maxseqlen 128 - 训练total step 62.5W - GPU RTX3090 - 训练时间总共耗费2.5天
All-time downloads
719
Public
Repo size
108 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin54.3 MB · 99%
At a glance
- Library
- transformers
- License
- mit
- Model type
- electra
- Access
- Public
- Created
- Mar 2, 2022
- Updated
- Sep 22, 2021
- SHA
- b8f43604
Datasets
Model card
From the Hugging Face model README
一、 个人在openwebtext数据集上训练得到的electra-small模型
二、 复现结果(dev dataset)
| Model | CoLA | SST | MRPC | STS | QQP | MNLI | QNLI | RTE | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Metrics | MCC | Acc | Acc | Spearman | Acc | Acc | Acc | Acc | |
| ELECTRA-Small-OWT(original) | 56.8 | 88.3 | 87.4 | 86.8 | 88.3 | 78.9 | 87.9 | 68.5 | 80.36 |
| ELECTRA-Small-OWT (this) | 55.82 | 89.67 | 87.0 | 86.96 | 89.28 | 80.08 | 87.50 | 66.07 | 80.30 |
三、 训练细节
- 数据集 openwebtext
- 训练batch_size 256
- 学习率lr 5e-4
- 最大句子长度max_seqlen 128
- 训练total step 62.5W
- GPU RTX3090
- 训练时间总共耗费2.5天
四、 使用
import torch
from transformers.models.electra import ElectraModel, ElectraTokenizer
tokenizer = ElectraTokenizer.from_pretrained("junnyu/electra_small_discriminator")
model = ElectraModel.from_pretrained("junnyu/electra_small_discriminator")
inputs = tokenizer("Beijing is the capital of China.", return_tensors="pt")
with torch.no_grad():
outputs = model(**inputs)
print(outputs[0].shape)
- Library
- transformers
- Type
- electra
- License
- mit
- Languages
- en
- Created
- Mar 2, 2022
- Updated
- Sep 22, 2021