Downloads · 30 days
0
RWKV7-0.1B-World2.8-SAE
fffffgggg54/RWKV7-0.1B-World2.8-SAE
RWKV7-0.1B-World2.8-SAE is a machine learning model from fffffgggg54. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
state sae (64 |- 8192 pick 16) 1 sae per head trying to reconstuct S @ ones(64,1)
Access
Public
Updated Nov 7, 2026
Repo size
10.3 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pth10.3 GB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Mar 19, 2025
- Updated
- Nov 7, 2026
- SHA
- c301886d
Base models
Datasets
Model card
From the Hugging Face model README
state sae (64 |-> 8192 pick 16) 1 sae per head trying to reconstuct S @ ones(64,1)
residual sae (768 |-> 32768 pick 64) sae on residual stream after every time-mix and channel-mix block
module sae (768 |-> 32768 pick 64) sae on the outputs of every time-mix and channel-mix block
training and reported losses are normalized MSE as defined in "Scaling and evaluating sparse autoencoders" by OpenAI
training code: https://github.com/fffffgggg54/StateSAE
- License
- apache-2.0
- Created
- Mar 19, 2025
- Updated
- Nov 7, 2026