Downloads · 30 days
261
1% of all-time downloads
deberta-v3-base-korean
team-lucid/deberta-v3-base-korean
deberta-v3-base-korean is a machine learning model from team-lucid. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.
DeBERTa는 Disentangled Attention과 Enhanced Masked Language Model을 통해 BERT의 성능을 향상시킨 모델입니다. 그중 DeBERTa V3은 ELECTRA-Style Pre-Training에 Gradient-Disentangled Embedding Sharing을 적용하여 DeBERTA를 개선했습니다.
All-time downloads
31.6K
Public
Parameters
135M
2.2 GB on disk
Likes
16
Public
Repo makeup
Click a slice to open those files.
.bin541 MB · 25%
Parameter types
How the weights are stored.
F32135M · 100%
Model card
From the Hugging Face model README
deberta-v3-base-korean
Model Details
DeBERTa는 Disentangled Attention과 Enhanced Masked Language Model을 통해 BERT의 성능을 향상시킨 모델입니다. 그중 DeBERTa V3은 ELECTRA-Style Pre-Training에 Gradient-Disentangled Embedding Sharing을 적용하여 DeBERTA를 개선했습니다.
이 연구는 구글의 TPU Research Cloud(TRC)를 통해 지원받은 Cloud TPU로 학습되었습니다.
How to Get Started with the Model
from transformers import AutoTokenizer, DebertaV2ForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("team-lucid/deberta-v3-base-korean")
model = DebertaV2ForSequenceClassification.from_pretrained("team-lucid/deberta-v3-base-korean")
inputs = tokenizer("안녕, 세상!", return_tensors="pt")
outputs = model(**inputs)
Evaluation
| Backbone<br/>Parameters(M) | NSMC<br/>(acc) | PAWS<br/>(acc) | KorNLI<br/>(acc) | KorSTS<br/>(spearman) | Question Pair<br/>(acc) | |
|---|---|---|---|---|---|---|
| DistilKoBERT | 22M | 88.41 | 62.55 | 70.55 | 73.21 | 92.48 |
| KoBERT | 85M | 89.63 | 80.65 | 79.00 | 79.64 | 93.93 |
| XLM-Roberta-Base | 85M | 89.49 | 82.95 | 79.92 | 79.09 | 93.53 |
| KcBERT-Base | 85M | 89.62 | 66.95 | 74.85 | 75.57 | 93.93 |
| KcBERT-Large | 302M | 90.68 | 70.15 | 76.99 | 77.49 | 94.06 |
| KoELECTRA-Small-v3 | 9.4M | 89.36 | 77.45 | 78.60 | 80.79 | 94.85 |
| KoELECTRA-Base-v3 | 85M | 90.63 | 84.45 | 82.24 | 85.53 | 95.25 |
| Ours | ||||||
| DeBERTa-xsmall | 22M | 91.21 | 84.40 | 82.13 | 83.90 | 95.38 |
| DeBERTa-small | 43M | 91.34 | 83.90 | 81.61 | 82.97 | 94.98 |
| DeBERTa-base | 86M | 91.22 | 85.5 | 82.81 | 84.46 | 95.77 |
* 다른 모델의 결과는 KcBERT-Finetune 과 KoELECTRA를 참고했으며, Hyperparameter 역시 다른 모델과 유사하게 설정습니다.
Model Memory Requirements
| dtype | Largest Layer or Residual Group | Total Size | Training using Adam |
|---|---|---|---|
| float32 | 187.79 MB | 513.77 MB | 2.01 GB |
| float16/bfloat16 | 93.9 MB | 256.88 MB | 1.0 GB |
| int8 | 46.95 MB | 128.44 MB | 513.77 MB |
| int4 | 23.47 MB | 64.22 MB | 256.88 MB |
- Library
- transformers
- Type
- deberta-v2
- License
- apache-2.0
- Languages
- ko
- Created
- Jun 30, 2023
- Updated
- Sep 16, 2024