SAEVerbalizer-27B

SAEVerbalizer-27B

SAEVerbalizer-27B is the default verbalizer from SAEVerbalizer: Generating Explanations for Sparse Autoencoder Features via Representation Verbalization. It is initialized from google/gemma-3-27b-it and fine-tuned to generate natural-language explanations from injected Sparse Autoencoder (SAE) decoder directions.

Resources

• Paper: arXiv:2608.13538
• Code: THU-KEG/SAEVerbalizer
• Evaluation data: THU-KEG/SAEVerbalizer-Data
• SAE: google/gemma-scope-2-27b-it

Usage

Run this checkpoint with the inference pipeline provided in the official code repository. The pipeline loads the corresponding SAE decoder direction and injects it into the model's residual stream before generation.

Configuration

• Verbalizer backbone: google/gemma-3-27b-it
• Injection layer: 16
• Training pairs: 48k
• Injection: norm-matched additive injection
• Injection coefficient: alpha = 0.2
• Injection span: final four tokens of the fixed verbalization prompt

This repository contains a full Transformers checkpoint. Inference also requires the layer-16 width-262k l0_medium SAE linked above.

License

SAEVerbalizer-27B is a modified Model Derivative of google/gemma-3-27b-it. The model weights were modified through SAEVerbalizer fine-tuning and are distributed under the Gemma Terms of Use. See the NOTICE file for the required attribution notice.

Citation

@article{meng2026saeverbalizer,
  title   = {SAEVerbalizer: Generating Explanations for Sparse Autoencoder Features via Representation Verbalization},
  author  = {Meng, Weihan and Guo, Hongzhu and Jing, Yi and Liu, Dewen and Yao, Zijun and Wang, Xiaozhi and Hou, Lei and Li, Juanzi},
  journal = {arXiv preprint arXiv:2608.13538},
  year    = {2026}
}