qwen36-27b-sae-l42-1b

BatchTopK SAE for Qwen/Qwen3.6-27B, layer 42 (resid_post)

A BatchTopK sparse autoencoder on the residual-stream output of decoder layer 42 of Qwen/Qwen3.6-27B (d_model=5120), trained with dictionary_learning so it loads directly.

Specs

How it compares to Adam Karvonen's Qwen3-8B SAEs

Same BatchTopK recipe (dictionary_learning BatchTopKTrainer), same aux-k dead-feature loss (auxk_alpha=1/32), same warmup + learned-threshold schedule. Differences: this SAE is on the 27B model (d=5120, layer 42), uses F=131072 / k=64, and was trained on ~250M tokens — about half the ~500M tokens Adam trains (his 8B batch_top_k is F=65536, k=80, ~500M tokens) on a 2x larger dictionary, so treat this as a research checkpoint (a longer run would likely lower the dead-feature rate). Method is otherwise identical.

Files

• saes_Qwen_Qwen3.6-27B_batch_top_k/resid_post_layer_42/trainer_0/ae.pt — SAE state_dict
• .../trainer_0/config.json — trainer/buffer config
• maxacts/acts_Qwen_Qwen3.6-27B_layer_42_trainer_0_context_length_32.pt — per-feature top-32 activating windows {"max_tokens":[F,N,32], "max_acts":[F,N,32]}
• data/examples.parquet — decoded top-activating text windows for every live feature (browse in the HF Data Studio)

Max-activating examples (sample)

The ⟦…⟧ marks the peak-activating token in each window. Full set for all live features is in data/examples.parquet.

Load (matches the eval loader)

import torch
p = torch.load("ae.pt")
km = {"encoder.weight":"W_enc","decoder.weight":"W_dec","encoder.bias":"b_enc","bias":"b_dec","b_dec":"b_dec"}
r = {km[k]: v for k, v in p.items() if k in km}
W_enc, W_dec = r["W_enc"].T.contiguous(), r["W_dec"].T.contiguous()   # [d,F], [F,d] (unit-norm rows)
b_enc, b_dec = r["b_enc"], r["b_dec"]
def encode(x):                       # x = RAW layer-42 residuals
    pre = torch.relu((x - b_dec) @ W_enc + b_enc)
    return pre * (pre > p["threshold"])