celeba-sae-top_k-64-patches_only-layer_6-hook_resid_post

CLIP Sparse Autoencoder Checkpoint

This model is a sparse autoencoder trained on CLIP's internal representations.

Model Details

Architecture

• Layer: 6
• Layer Type: hook_resid_post
• Model: open-clip:laion/CLIP-ViT-B-32-DataComp.XL-s13B-b90K
• Dictionary Size: 49152
• Input Dimension: 768
• Expansion Factor: 64
• CLS Token Only: False

Training

• Training Images: 324085
• Learning Rate: 0.0001
• L1 Coefficient: 0.0002
• Batch Size: 4096
• Context Size: 49

Performance Metrics

Sparsity

• L0 (Active Features): 256.0000
• Dead Features: 0
• Mean Log10 Feature Sparsity: -2.8938
• Features Below 1e-5: 0.0000
• Features Below 1e-6: 0.0000
• Mean Passes Since Fired: 0.1934

Reconstruction

• Explained Variance: 0.8451
• Explained Variance Std: 0.0546
• MSE Loss: 0.0014
• L1 Loss: 0
• Overall Loss: 0.0014

Training Details

• Training Duration: 1165 seconds
• Final Learning Rate: 0.0000
• Warm Up Steps: 500
• Gradient Clipping: 1

Additional Information

• Original Checkpoint Path: /network/scratch/p/praneet.suresh/celeba_checkpoints/34f0853e-tinyclip_sae_16_hyperparam_sweep_lr/n_images_324169.pt
• Wandb Run: https://wandb.ai/perceptual-alignment/celeba-sweep-topk-patches_all_layers/runs/drz4z0jz
• Random Seed: 42