Downloads · 30 days
3
6% of all-time downloads
fw-tiny-old
tdooms/fw-tiny-old
fw-tiny-old is a machine learning model from tdooms. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.
This is the tiny version of the bilinear transformers trained on FineWeb-edu. The primary purpose of this model is interpretability, most design choices were made with that in mind.
All-time downloads
50
Public
Parameters
125M
499 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors499 MB · 100%
At a glance
- Library
- transformers
- Access
- Public
- Created
- Oct 6, 2024
- Updated
- Oct 18, 2024
- SHA
- f4360a86
Model card
From the Hugging Face model README
FW Tiny
This is the tiny version of the bilinear transformers trained on FineWeb-edu. The primary purpose of this model is interpretability, most design choices were made with that in mind.
The code to run this custom model can be found here, along with many utility functions for weight-based interpretability.
Model Details
- 125 million parameters
- 8 layers
- 12 attention heads
- model dimension 768
- bilinear MLP with expansion factor 4
- context length of 512
- trained for 16B tokens
- rotary positional embedding
- Mixtral tokenizer
- Library
- transformers
- Created
- Oct 6, 2024
- Updated
- Oct 18, 2024