Downloads · 30 days
9
2% of all-time downloads
structformer_s1_final_with_pos
omarmomen/structformer_s1_final_with_pos
structformer_s1_final_with_pos is a fill-mask model from omarmomen. Use it when you need the model to fill a missing word. It is set up for transformers. The card lists the license as mit.
This model is part of the experiments in the published paper at the BabyLM workshop in CoNLL 2023. The paper titled "Increasing The Performance of Cognitively Inspired Data-Efficient Language Models via Implicit Struc…
All-time downloads
446
Public
Repo size
31 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin21.4 GB · 54%
At a glance
- Task
- Fill-Mask
- Library
- transformers
- License
- mit
- Model type
- structformer
- Access
- Public
- Created
- Oct 12, 2023
- Updated
- Mar 26, 2024
- SHA
- 8952578b
Try a prompt
Datasets
Model card
From the Hugging Face model README
Model Card for omarmomen/structformer_s1_final_with_pos
This model is part of the experiments in the published paper at the BabyLM workshop in CoNLL 2023. The paper titled "Increasing The Performance of Cognitively Inspired Data-Efficient Language Models via Implicit Structure Building" (https://aclanthology.org/2023.conll-babylm.29/)
<strong>omarmomen/structformer_s1_final_with_pos</strong> is a modification of the vanilla transformer encoder to incorporate syntactic inductive bias using an unsupervised parsing mechanism.
This model variant places the parser network ahead of all the attention blocks.
The model is pretrained on the BabyLM 10M dataset using a custom pretrained RobertaTokenizer (https://huggingface.co/omarmomen/babylm_tokenizer_32k).
- Task
- Fill-Mask
- Library
- transformers
- Type
- structformer
- License
- mit
- Languages
- en
- Created
- Oct 12, 2023
- Updated
- Mar 26, 2024