HDT-E

Model Weights Comming Soon!

Using HDT

To use the pre-trained model for masked language modeling, use the following snippet:

from transformers import AutoModelForMaskedLM, AutoTokenizer

# See the `MDLM` collection page on the hub for list of available models.
tokenizer = transformers.AutoTokenizer.from_pretrained('howey/HDT-E')
model_name = 'howey/HDT-E'
model = AutoModelForMaskedLM.from_pretrained(model_name)

For more details, please see our github repository: HDT

Model Details

The model, which has a context length of 8192 and is similar in size to BERT with approximately 110M parameters, was trained on standard masked language modeling task with a Transformer-based architecture using our proposed hierarchical attention. The training regimen comprised 24 hours on the ArXiv+Wikipedia+HUPD corpus, involving the processing of a total of 1.3 billion tokens.

For more details, please see our paper: HDT: Hierarchical Document Transformer.

Citation

<!-- If there is a paper or blog post introducing the model, the Bibtex information for that should go in this section. -->

Please cite our work using the bibtex below:

BibTeX:

@inproceedings{He2024COLM,
      title={HDT: Hierarchical Document Transformer},
      author={Haoyu He and Markus Flicke and Jan Buchmann and Iryna Gurevych and Andreas Geiger},
      year={2024},
      booktitle={Conference on Language Modeling}
}

Model Card Contact

Haoyu ([email protected])