Downloads · 30 days
0
Aleph-Alpha/tfree-research-vocab-32k-fineweb-steps-370k-instruct-10k
tfree-research-vocab-32k-fineweb-steps-370k-instruct-10k is a machine learning model from Aleph-Alpha. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as other.
This model is to support our ongoing research T-Free. It is publicly available under the Open Aleph License, a license explicitly allowing for non-commercial research and educational use.
Downloads · 30 days
0
Access
Public
Updated Aug 29, 2024
Repo size
19.7 GB
Likes
1
Public
Click a slice to open those files.
.pt19.7 GB · 100%
From the Hugging Face model README
This model is to support our ongoing research T-Free. It is publicly available under the Open Aleph License, a license explicitly allowing for non-commercial research and educational use.
The model was trained on 1 epoch of the fineweb-edu dataset with a sequence length of 4k and 1k batch-size. It was further continued with a llama-style instruction finetuning.
It has an embedding layer of dimension 32k, vocab population of 10 (activations per trigram) and partial lower case overlap of 2 (i.e. 2 of the 10 activations overlap with the trigrams lowercase counterpart). This model aggregates all activations by taking the sum only.
Minimal Example:
# download checkpoint from huggingface
apt-get install git-lfs
git clone https://huggingface.co/Aleph-Alpha/tfree-research-vocab-32k-fineweb-steps-370k-instruct-10k
# install this tfree repository
git clone https://github.com/Aleph-Alpha/trigrams.git
pip install -e trigrams
# adjust checkpoint path in inference.py, then run it
python inference.py