Downloads · 30 days
13
1% of all-time downloads
awni00/DAT-sa8-ra8-ns1024-sh8-nkvh4-343M
DAT-sa8-ra8-ns1024-sh8-nkvh4-343M is a text generation model from awni00. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
This is a Dual-Attention Transformer Language Model, trained on the fineweb-edu dataset. The model is 343M parameters.
Downloads · 30 days
13
1% of all-time downloads
All-time downloads
2.3K
Public
Parameters
344M
8.3 GB on disk
Likes
0
Public
Click a slice to open those files.
.pt4.1 GB · 75%
From the Hugging Face model README
This is a Dual-Attention Transformer Language Model, trained on the fineweb-edu dataset. The model is 343M parameters.
| Size | Training Tokens | Layers | Model Dimension | Self-Attention Heads | Relational Attention Heads | Relation Dimension | Context Length |
|---|---|---|---|---|---|---|---|
| 343M | 10B | 24 | 1024 | 8 | 8 | 8 | 1024 |
Use the code below to get started with the model. First, install the dual-attention python package hosted on PyPI via pip install dual-attention.
To load directly from huggingface hub, use the HFHub wrapper.
from dual_attention.hf import DualAttnTransformerLM_HFHub
DualAttnTransformerLM_HFHub.from_pretrained('awni00/DAT-sa8-ra8-ns1024-sh8-nkvh4-343M')
The model was trained using the following setup:
For more detailed training information, please refer to the paper.
See paper.
The DAT-LM-Visualization app is built to visualize the representations learned in a Dual Attention Transformer language model. It is hosted on Huggingface spaces using their free CPU resources. You can select a pre-trained DAT-LM model, enter a prompt, and visualize the internal representations in different parts of the model. You can also run the app locally (e.g., to use your own GPU) via the PyPI package.
Also, see paper.
@misc{altabaa2024disentanglingintegratingrelationalsensory,
title={Disentangling and Integrating Relational and Sensory Information in Transformer Architectures},
author={Awni Altabaa and John Lafferty},
year={2024},
eprint={2405.16727},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2405.16727},
}