safety-dictionary-topk

Safety Dictionary Top-K

This repository contains the sparse autoencoders (SAEs) trained for the paper Do LLMs Know Their Vulnerable Scenarios?, also available on Hugging Face Papers.

Training Details

• Architecture: TopK SAE with a dictionary width of 65,536 and k = 64, trained on residual-stream (resid_post) activations.
• Training data: A 50/50 mixture of SlimPajama and BeaverTails. BeaverTails prompts are paired with responses generated by the corresponding target model and formatted with its chat template.
• Activation mixture: Each batch contains 50% Base-model activations and 50% Instruct-model activations, randomly shuffled to train one shared dictionary for both models.
• Training scale: Each SAE is trained in bfloat16 on 500 million tokens (approximately 244,140 steps with an SAE batch size of 2,048) using a context length of 4,096.
• Layers: Llama-3.1-8B-Mixed uses layers 27 and 31; Ministral-3-8B-Mixed uses layers 29 and 33; Qwen3.5-9B-Mixed uses layers 27 and 31.