Downloads · 30 days
97
1% of all-time downloads
SoLU_2L_v10_old
NeelNanda/SoLU_2L_v10_old
SoLU_2L_v10_old is a machine learning model from NeelNanda. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.
A 2L, width 736 SoLU model trained on 15B tokens of the Pile. Bugs: the layernorm just before the unembed is an RMS norm, and the width is not a multiple of 64, so dhead=64 and nheads=11, and nheads dhead != dmodel :(
All-time downloads
11.8K
Public
Repo size
54.3 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pth54.3 GB · 100%
At a glance
- Library
- transformers
- Access
- Public
- Created
- Oct 12, 2022
- Updated
- Oct 26, 2022
- SHA
- 9f9ca203
Model card
From the Hugging Face model README
A 2L, width 736 SoLU model trained on 15B tokens of the Pile. Bugs: the layernorm just before the unembed is an RMS norm, and the width is not a multiple of 64, so d_head=64 and n_heads=11, and n_heads * d_head != d_model :(
- Library
- transformers
- Created
- Oct 12, 2022
- Updated
- Oct 26, 2022