Downloads · 30 days
3
14% of all-time downloads
tatsatx/llama2_7b_hf_sae_layers4_10_16_22_28_k32_latents400_wikitext103
llama2_7b_hf_sae_layers4_10_16_22_28_k32_latents400_wikitext103 is a machine learning model from tatsatx. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
This repository contains Sparse Autoencoder (SAE) models trained on the meta-llama/Llama-2-7b-hf model using WikiText-103 dataset.
Downloads · 30 days
3
14% of all-time downloads
All-time downloads
22
Public
Repo size
33.4 MB
Likes
0
Public
Click a slice to open those files.
.pt33.4 MB · 100%
From the Hugging Face model README
This repository contains Sparse Autoencoder (SAE) models trained on the meta-llama/Llama-2-7b-hf model using WikiText-103 dataset.
meta-llama/Llama-2-7b-hf| Layer | Loss Recovered | L0 Sparsity | Dead Features | Health Status |
|---|---|---|---|---|
| 4 | 98.42% (WikiText)<br>98.51% (SQuAD) | 61.37-63.35 | 28.25% (WikiText)<br>18.00% (SQuAD) | ⚠️ 3/4 Healthy |
| 10 | 96.84% (WikiText)<br>96.68% (SQuAD) | 58.80-58.61 | 1.00% (WikiText)<br>0.75% (SQuAD) | ✅ 4/4 Healthy |
| 16 | 90.02% (WikiText)<br>89.83% (SQuAD) | 76.38-76.00 | 0.00% | ✅ 4/4 Healthy |
| 22 | 66.24% (WikiText)<br>65.95% (SQuAD) | 94.33-94.42 | 0.00% | ✅ 4/4 Healthy |
| 28 | 4.87% (WikiText)<br>8.11% (SQuAD) | 125.58-123.13 | 0.00% | ⚠️ 3/4 Healthy |
llama2_7b_hf_sae_layers4_10_16_22_28_k32_latents400_wikitext103/
├── config.json # Model configuration
├── state.pt # Training state
├── optimizer_0.pt # Optimizer state
├── lr_scheduler_0.pt # Learning rate scheduler
├── rank_0_state.pt # Distributed training state
├── layers.4/ # Layer 4 SAE model
├── layers.10/ # Layer 10 SAE model
├── layers.16/ # Layer 16 SAE model
├── layers.22/ # Layer 22 SAE model
└── layers.28/ # Layer 28 SAE model
from transformers import AutoModel
import torch
# Load a specific layer SAE model
layer_model = AutoModel.from_pretrained("tatsatx/llama2_7b_hf_sae_layer16_k32_latents400_wikitext103")
# Load the complete multi-layer model
complete_model = AutoModel.from_pretrained("tatsatx/llama2_7b_hf_sae_layers4_10_16_22_28_k32_latents400_wikitext103")
jdb0zp9wThese models are part of ongoing research in interpretability and sparse autoencoders. For questions or collaborations, please refer to the main research repository.
This model follows the license of the base Llama-2-7B model. Please refer to Meta's licensing terms for commercial use.
Model trained on 2025-08-31 using SAE methodology on Llama-2-7B base model Performance metrics validated across WikiText-103 and SQuAD datasets