Downloads · 30 days
3
14% of all-time downloads
tatsatx/llama2_7b_finance_sae_layers4_10_16_22_28_k32_latents400_wikitext103
llama2_7b_finance_sae_layers4_10_16_22_28_k32_latents400_wikitext103 is a machine learning model from tatsatx. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
This repository contains Sparse Autoencoder (SAE) models trained on the cxllin/Llama2-7b-Finance model using WikiText-103 dataset.
Downloads · 30 days
3
14% of all-time downloads
All-time downloads
22
Public
Repo size
33.4 MB
Likes
0
Public
Click a slice to open those files.
.pt33.4 MB · 100%
From the Hugging Face model README
This repository contains Sparse Autoencoder (SAE) models trained on the cxllin/Llama2-7b-Finance model using WikiText-103 dataset.
cxllin/Llama2-7b-Finance (Financial fine-tuned from meta-llama/Llama-2-7b-hf)| Layer | Loss Recovered | L0 Sparsity | Dead Features | Health Status |
|---|---|---|---|---|
| 4 | 56.146% (WikiText) | 55.112 | 85.750% | ❌ 2/4 Healthy |
| 10 | 58.234% (WikiText) | 48.567 | 78.450% | ❌ 2/4 Healthy |
| 16 | 62.891% (WikiText) | 42.123 | 65.320% | ⚠️ 3/4 Healthy |
| 22 | 59.456% (WikiText) | 51.789 | 72.180% | ❌ 2/4 Healthy |
| 28 | 55.678% (WikiText) | 54.321 | 89.120% | ❌ 1/4 Healthy |
llama2_7b_finance_layers4_10_16_22_28_k32_latents400_wikitext103_torchrun/
├── config.json # Model configuration
├── state.pt # Training state
├── optimizer_0.pt # Optimizer state
├── lr_scheduler_0.pt # Learning rate scheduler
├── rank_0_state.pt # Distributed training state
├── layers.4/ # Layer 4 SAE model
├── layers.10/ # Layer 10 SAE model
├── layers.16/ # Layer 16 SAE model
├── layers.22/ # Layer 22 SAE model
└── layers.28/ # Layer 28 SAE model
from transformers import AutoModel
import torch
# Load a specific layer SAE model
layer_model = AutoModel.from_pretrained("tatsatx/llama2_7b_finance_sae_layer16_k32_latents400_wikitext103")
# Load the complete multi-layer model
complete_model = AutoModel.from_pretrained("tatsatx/llama2_7b_finance_sae_layers4_10_16_22_28_k32_latents400_wikitext103")
| Metric | FinLLama-7B | Base Llama-2-7B | Difference |
|---|---|---|---|
| Healthy Layers | 0/5 (0%) | 3/5 (60%) | -60% |
| Avg Loss Recovery | 58.48% | 71.1% | -12.62% |
| Dead Features | 65.32-89.12% | 0.0-28.25% | Significantly Higher |
| L0 Sparsity Range | 42.12-55.11 | 58.61-125.58 | More Consistent |
These models are part of ongoing research in interpretability and sparse autoencoders for domain-specific models. For questions or collaborations, please refer to the main research repository.
This model follows the license of the base Llama-2-7B model and cxllin/Llama2-7b-Finance. Please refer to the respective licensing terms for commercial use.
Model trained on 2025-08-31 using SAE methodology on FinLLama-7B fine-tuned model Performance metrics validated across WikiText-103 dataset Comparative analysis with base Llama-2-7B SAE models