Downloads · 30 days
58
2% of all-time downloads
longformer-base-8192
jeohalves/longformer-base-8192
longformer-base-8192 is a machine learning model from jeohalves. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.
Longformer is a transformer model for long documents.
All-time downloads
3.2K
Public
Parameters
152M
1.2 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin607 MB · 50%
At a glance
- Library
- transformers
- License
- apache-2.0
- Model type
- longformer
- Access
- Public
- Created
- Jul 23, 2024
- Updated
- Nov 7, 2024
- SHA
- 291f8995
Model card
From the Hugging Face model README
longformer-base-8192
Longformer is a transformer model for long documents.
longformer-base-8192 is a BERT-like model started from the RoBERTa checkpoint and pretrained for MLM on long documents. It supports sequences of length up to 8,192.
Longformer uses a combination of a sliding window (local) attention and global attention. Global attention is user-configured based on the task to allow the model to learn task-specific representations.
Please refer to the examples in modeling_longformer.py and the paper for more details on how to set global attention.
Citing
If you use Longformer in your research, please cite Longformer: The Long-Document Transformer.
@article{Beltagy2020Longformer,
title={Longformer: The Long-Document Transformer},
author={Iz Beltagy and Matthew E. Peters and Arman Cohan},
journal={arXiv:2004.05150},
year={2020},
}
Longformer is an open-source project developed by the Allen Institute for Artificial Intelligence (AI2).
AI2 is a non-profit institute with the mission to contribute to humanity through high-impact AI research and engineering.
- Library
- transformers
- Type
- longformer
- License
- apache-2.0
- Languages
- en
- Created
- Jul 23, 2024
- Updated
- Nov 7, 2024