Downloads · 30 days
21
2% of all-time downloads
hierarchical-dialog-bert
igorktech/hierarchical-dialog-bert
hierarchical-dialog-bert is a fill-mask model from igorktech. Use it when you need the model to fill a missing word. It is set up for transformers. The card lists the license as cc-by-nc-sa-4.0.
This model is based on An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification. Ilias Chalkidis, Xiang Dai, Manos Fergadiotis, Prodromos Malakasiotis, and Desmond Elliott. 2022…
All-time downloads
1.3K
Public
Repo size
140 MB
Likes
2
Public
Repo makeup
Click a slice to open those files.
.bin69.8 MB · 99%
At a glance
- Task
- Fill-Mask
- Library
- transformers
- License
- cc-by-nc-sa-4.0
- Model type
- hierarchical-transformer
- Access
- Public
- Created
- Jun 6, 2023
- Updated
- Aug 16, 2023
- SHA
- b4de9a8c
Try a prompt
Datasets
Model card
From the Hugging Face model README
Model description
This model is based on An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification. Ilias Chalkidis, Xiang Dai, Manos Fergadiotis, Prodromos Malakasiotis, and Desmond Elliott. 2022. arXiv:2210.05529 (Preprint).
Initial weights were taken from google/bert_uncased_L-8_H-256_A-4. Model was additionally pretrained for 20_000 steps on 5m lines of text from english version of OpenSubtitles dataset.
Maximum input length is 512 tokens that is enoungh to encode dialog with few previous utterances (average sentence length per utterance in SWDA, MAPTASK, MRDA, BT_OASIS, FRAMES, AMI, DSTC3 is less than 11 tokens).
- Task
- Fill-Mask
- Library
- transformers
- Type
- hierarchical-transformer
- License
- cc-by-nc-sa-4.0
- Languages
- en
- Created
- Jun 6, 2023
- Updated
- Aug 16, 2023