mi-steering-denoiser-gpt2

Denoising Activation Steering in GPT-2

TLAB Mechanistic Interpretability submission.
A lightweight denoising approach for reducing language degradation caused by activation steering, plus an orthogonal correction that preserves the intended steering component.

<p align="center"> <img src="figures/negative_pareto.png" width="49%" /> <img src="figures/positive_pareto.png" width="49%" /> </p>

TL;DR

Standard activation steering modifies a residual-stream activation as

[ \tilde h = h + \alpha v, ]

where v is a steering direction and alpha controls strength. Strong interventions can push the representation away from the model's usual activation manifold, leading to repetition and degeneration.

I train a small residual MLP denoiser on clean GPT-2 activations corrupted with Gaussian noise, without exposing it to the validation steering vectors:

[ \hat h = D(h + \epsilon), \qquad \epsilon \sim \mathcal N(0,\sigma^2 I). ]

At inference I compare:

1. Ordinary steering [ h' = h + \alpha v ]

2. Full denoising [ h' = D(h + \alpha v) ]

3. Orthogonal denoising [ \delta = D(h+\alpha v) - (h+\alpha v) ] [ \delta_\perp = \delta - \frac{\delta \cdot v}{|v|^2}v ] [ h' = h + \alpha v + \delta_\perp. ]

The third method is motivated by a mechanistic observation: the denoiser correction increasingly points against the steering vector as steering becomes stronger. Removing only the parallel correction preserves the intended semantic intervention while keeping the orthogonal cleanup.

---

Main findings

• The denoiser reduces validation reconstruction MSE from 0.4895 to 0.3322 — a 32.1% reduction.
• The denoiser was trained on clean activations only; the validation steering directions were not seen during training.
• For negative-sentiment steering, the correction becomes increasingly anti-aligned with the steering direction: cos(δ, v) changes from about -0.34 at r=0.10 to -0.71 at r=0.50.
• This motivates the orthogonal correction.
• At moderate steering strengths, denoising-based methods improve parts of the concept–quality Pareto frontier.
• The effect generalizes from the negative steering direction to a held-out positive-sentiment SAE feature.
• The improvement is not universal: extreme steering still causes degeneration, and the full denoiser can alter clean behavior even at r=0.

---

Experimental setup

Relative steering strength is normalized by activation magnitude:

[ r = \frac{|\alpha v|_2}{\mathbb E|h|_2}, \qquad \alpha = r\frac{\mathbb E|h|_2}{|v|_2}. ]

The measured mean activation norm used in the experiments was approximately 79.20.

---

Steering directions

The directions are SAE decoder columns and were first validated with ordinary activation steering.

The scientific feature was not included in the final quantitative tables, so no quantitative generalization claim is made for it.

---

Denoiser training

The objective is

[ \mathcal L = |h - D(h+\epsilon)|_2^2, \qquad \epsilon \sim \mathcal N(0, 0.7^2 I). ]

<p align="center"> <img src="figures/denoiser_mse.png" width="62%" /> </p>

The identity/noisy-input baseline gives validation MSE ≈ 0.4895.
The best checkpoint reaches validation MSE ≈ 0.3322 at epoch 15.

This is useful as a training sanity check, but reconstruction MSE is not the main success criterion. The main question is whether steering achieves a better trade-off between concept strength and generation quality.

---

Why orthogonal correction?

For

[ x=h+\alpha v, \qquad \delta=D(x)-x, ]

the denoiser correction has a substantial component parallel to v, and that component is negative at stronger steering.

<p align="center"> <img src="figures/correction_alignment.png" width="68%" /> </p>

Measured on the negative-sentiment direction:

| r | ||αv|| | ||δ|| | cos(δ,v) | |---:|---:|---:|---:| | 0.10 | 8.25 | 9.92 | -0.341 | | 0.20 | 16.50 | 11.09 | -0.525 | | 0.25 | 20.63 | 11.72 | -0.582 | | 0.30 | 24.75 | 12.35 | -0.625 | | 0.40 | 33.01 | 13.53 | -0.679 | | 0.50 | 41.26 | 14.54 | -0.710 |

At r=0.25, for example, the correction norm is ≈ 11.72, with ≈ 6.83 in the parallel component and ≈ 9.53 in the perpendicular component.

This suggests that full denoising partly undoes the intended steering. Orthogonal correction removes only the component of the denoiser update parallel to v.

---

Evaluation

Concept score

For sentiment steering I use the pretrained SST-2 classifier:

• negative steering: [ C_- = P(\text{NEGATIVE}\mid\text{continuation}) ]

• positive steering: [ C_+ = P(\text{POSITIVE}\mid\text{continuation}) ]

Higher is better.

Language-quality diagnostics

I report:

• token-level NLL,
• Dist-1,
• Dist-2,
• Dist-3,
• qualitative generations.

Important: Dist-n is treated as a repetition/diversity proxy, not a complete fluency metric. NLL is also imperfect: repetitive text can be highly predictable and therefore have deceptively low NLL.

A local Qwen2.5-3B-Instruct judge was sanity-checked on fluent and deliberately degraded examples, but it was not used for the final aggregate results, because the raw generations were not preserved after a kernel restart. I therefore do not present LLM-judge scores as experimental evidence.

Why there are no TP / FP / TN / FN counts

This experiment does not define a binary classification task. The concept metric is a continuous probability and no classification threshold was preregistered. Reporting true positives / false negatives post hoc would therefore be misleading. The correct comparison is the continuous concept–quality trade-off.

---

Results: negative sentiment

<p align="center"> <img src="figures/negative_pareto.png" width="74%" /> </p>

Selected points:

Two useful Pareto-style comparisons:

• orthogonal r=0.20 vs baseline r=0.30: slightly higher concept score and higher Dist-3.
• orthogonal r=0.25 vs baseline r=0.40: higher concept score and substantially higher Dist-3.

At r=0.50, orthogonal correction no longer behaves consistently; extreme steering remains a failure regime.

---

Results: held-out positive sentiment

<p align="center"> <img src="figures/positive_pareto.png" width="74%" /> </p>

The denoiser was not trained on this validation direction.

Selected observations:

The held-out positive direction supports the same high-level conclusion: denoising-based correction can improve the trade-off in the moderate-steering regime, but neither full nor orthogonal correction universally dominates.

---

Example causal steering behavior

Positive feature 15385:

r = 0.00
"The experience was a bit different ..."

r = 0.20
"The experience was a success ... I was very impressed ..."

r = 0.25
"The experience was a success ... It was a great product ..."

r = 0.50
"The experience was described ... very good quality ...
 the quality of the service ... the service of the service ..."

This illustrates the characteristic steering curve: useful concept control at moderate strength, followed by repetition/degeneration at stronger intervention.

---

Repository structure

MI-steering/
├── checkpoints/
│   └── denoiser_v1.pt
├── figures/
│   ├── negative_pareto.png
│   ├── positive_pareto.png
│   ├── correction_alignment.png
│   └── denoiser_mse.png
├── notebooks/
│   ├── steering.ipynb
│   └── generalization.ipynb
├── scripts/
│   ├── collect_activations.py
│   └── train_denoiser.py
├── src/
│   └── denoiser.py
├── README.md
└── requirements.txt

---

Reproduction

Install dependencies:

pip install -r requirements.txt

Collect clean residual-stream activations:

python scripts/collect_activations.py

Train the denoiser:

python scripts/train_denoiser.py

The evaluation notebooks contain steering-direction validation, baseline steering, full denoising, orthogonal correction, metrics, and held-out positive-direction evaluation.

---

Best checkpoint

Best checkpoint:

checkpoints/denoiser_v1.pt

Validation MSE:

0.3322103939

Public Hugging Face repository:

https://huggingface.co/<YOUR_USERNAME>/mi-steering-denoiser-gpt2

Replace <YOUR_USERNAME> with the final public repository owner before submission.

---

Limitations

• GPT-2 Small only.
• One fixed intervention layer.
• Small validation prompt set.
• Greedy decoding only.
• Dist-n measures diversity/repetition rather than full linguistic fluency.
• NLL can reward predictable repetition.
• Full denoising changes model behavior even at r=0.
• The orthogonal method does not uniformly dominate full denoising.
• Extreme steering remains unstable.
• The scientific-discourse vector was causally validated only qualitatively.
• No final aggregate LLM-judge evaluation was completed.

---

Conclusion

A cheap denoiser trained only on noisy clean activations can improve activation steering without learning the held-out steering vector itself.

The most informative mechanistic result is that the learned denoising update increasingly opposes the steering direction at larger intervention strengths. Projecting that correction orthogonally to the steering vector gives a simple way to preserve the intended semantic intervention while still applying denoising in the remaining residual-stream directions.

The result is not a universal fix, but it produces clear Pareto-improving points in the moderate-steering regime and generalizes to a held-out positive sentiment direction.