Downloads · 30 days
254
7% of all-time downloads
longformer-base-4096-extra.pos.embd.only
allenai/longformer-base-4096-extra.pos.embd.only
longformer-base-4096-extra.pos.embd.only is a machine learning model from allenai. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.
This model is similar to longformer-base-4096 but it was pretrained to preserve RoBERTa weights by freezing all RoBERTa weights and only train the additional position embeddings.
All-time downloads
3.5K
Public
Repo size
2 GB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.h5765 MB · 56%
At a glance
- Library
- transformers
- Model type
- longformer
- Access
- Public
- Created
- Mar 2, 2022
- Updated
- Oct 3, 2022
- SHA
- fa20d443
Model card
From the Hugging Face model README
longformer-base-4096-extra.pos.embd.only
This model is similar to longformer-base-4096 but it was pretrained to preserve RoBERTa weights by freezing all RoBERTa weights and only train the additional position embeddings.
Citing
If you use Longformer in your research, please cite Longformer: The Long-Document Transformer.
@article{Beltagy2020Longformer,
title={Longformer: The Long-Document Transformer},
author={Iz Beltagy and Matthew E. Peters and Arman Cohan},
journal={arXiv:2004.05150},
year={2020},
}
Longformer is an open-source project developed by the Allen Institute for Artificial Intelligence (AI2).
AI2 is a non-profit institute with the mission to contribute to humanity through high-impact AI research and engineering.
- Library
- transformers
- Type
- longformer
- Languages
- en
- Created
- Mar 2, 2022
- Updated
- Oct 3, 2022