BrainPatch-Qwen2.5-1.5B

BrainPatch — Qwen2.5-1.5B-Instruct

A Top-K sparse autoencoder over layer 18 of a frozen Qwen2.5-1.5B-Instruct, plus the runtime for injecting its feature directions back into the residual stream.

GitHub: https://github.com/09Catho/BrainPatch

---

Read this first

This repository contains verified infrastructure with a negative behavioural result.

The pipeline works end to end and is reproducible. The behavioural claim does not hold: in the smoke_v0 intervention experiment, steering the selected SAE feature moved the model's output away from baseline — but a scale-matched random direction of identical L2 norm moved it further.

There is no evidence here that any feature direction carries specific behavioural meaning. That is why the shipped patches are named experimental-feature-727.json and not anti-sycophancy.json.

---

What this is

BrainPatch applies small activation-space interventions to a frozen language model. Nothing is fine-tuned; the base weights are never touched.

frozen LLM
   +  residual-stream activations   (layer 18, post-block)
   +  Top-K sparse autoencoder      (d_sae 2048, k 32)
   +  feature directions            (unit-norm decoder columns)
   +  runtime intervention hooks    (add / ablate, with schedules)

An intervention adds, at the hooked layer:

delta_raw = strength × unit_decoder_column / input_scale

input_scale normalises activations so E[||x||₂] = √d_in. Dividing by it maps the direction back to the raw residual stream, which is what makes strength mean the same physical thing across SAEs. For this SAE, input_scale = 0.5610531069008018.

What this is not

• Not evidence that SAE features correspond to human concepts.
• Not a claim that steering demonstrates beliefs, intentions, or any mental property.
• Not a source of validated behavioural labels. Every feature description is a hypothesis and is marked as one.
• Not free of side effects. Interventions may affect unrelated capabilities; the smoke test observed 9/10 vs 8/10 on ten hand-written probes, but that sample is far too small to establish degradation.
• Not portable to other models, layers, or SAEs. The format refuses mismatched application.

---

Contents

The Qwen base weights are not duplicated here. Load them from Qwen/Qwen2.5-1.5B-Instruct at revision 989aa7980e4cf806f80c7fef2b1adb7bc71aa306.

Raw activation shards are not published: 58.8 MB derived from a CC BY-SA corpus, fully reproducible from the recorded config.

---

Usage

Most users want the portable patch, not the SAE. A .brainpatch file is 6.4 KB and self-contained; the 72 MB SAE checkpoint is a research artifact you only need if you are discovering new features.

pip install "brainpatch[transformers,hub]"
brainpatch install 09Catho/BrainPatch-Qwen2.5-1.5B:patches/runtime/experimental-feature-727.brainpatch
brainpatch compare --model Qwen/Qwen2.5-1.5B-Instruct --patch experimental-feature-727 --prompt "Explain why the sky is blue."

from brainpatch import BrainPatchedModel

model = BrainPatchedModel.from_pretrained(
    "Qwen/Qwen2.5-1.5B-Instruct",
    revision="989aa7980e4cf806f80c7fef2b1adb7bc71aa306",
    backend="transformers",
    device="auto",
)
patch = model.install("experimental-feature-727")   # installed above
patch.strength = 1.0                                 # live, clamped to the patch envelope
print(model.generate("Solve this problem: what is 17 + 25?"))

patch.strength = 0.0 recovers baseline byte-identically — verified on an L4 with 0 applied hook passes.

Token-level schedule (Transformers backend only):

patch.schedule = {0: 0.0, 24: 1.0, 48: 2.0}

No Modal, no hosted service, and no network once the model and patch are local.

Verified runtime properties

Measured by modal run modal_app/app.py::test_transformers_backend on Qwen2.5-1.5B-Instruct / NVIDIA L4:

Compilation is numerically exact: the 6.4 KB portable artifact reproduces the original SAE-based pipeline's delta norm (28.5178) to fp16 rounding.

Overhead on L4 (3 runs, 96 tokens): −1.3% (within noise), 0.01 MB VRAM, 0.21 s patch load.

---

Modal reproduction

Every experiment ran on Modal on a single NVIDIA L4. No local GPU is required, and no model weights or activations are ever downloaded to a developer machine.

modal run modal_app/app.py::smoke_pipeline

Stage by stage:

modal run modal_app/app.py::cache_model
modal run modal_app/app.py::extract_activations --experiment smoke_v0 --target-tokens 20000
modal run modal_app/app.py::train_sae --experiment smoke_v0 --d-sae 2048 --k 32 --epochs 60
modal run modal_app/app.py::analyze_features --experiment smoke_v0
modal run modal_app/app.py::intervention_experiment --experiment smoke_v0 --strength 16

---

Experimental results

All figures below were measured. None are estimates.

Environment

Extraction

20,000 tokens from layer 18 (residual_post), sequence length 256, in 8.367 s → 2390.4 tokens/s, at 3084.01 bytes/token (= 1536 × 2 bf16 + 12 bytes int32 metadata). Peak VRAM 3553.4 MB.

Position 0 is excluded because it exhibited an extreme residual-stream activation outlier: measured norm 11052 against a corpus mean of ~70 at layer 18, a factor of 156. Including it would dominate the input-scale normalisation.

Such first-token outliers are commonly attributed to attention-sink behaviour, which is a plausible explanation here — but no attention weights were measured, so the mechanism is not established by these results.

SAE training

The 0.104 train/validation explained-variance gap is genuine overfitting. 19,000 training rows against a 2048-feature dictionary is roughly 9 rows per feature. Zero dead features at this scale is not a sign of health either. smoke_v0 exists to prove the pipeline, not to produce a good SAE.

L0 is bounded by k, not identically k. torch.topk always returns k indices, but the ReLU in front of it means a selected value can be zero whenever a token has fewer than k positive pre-activations. Reconstruction is unaffected (zeros scatter into a zeros tensor) and reported L0 counts only positive entries, so it is correct. Liveness accounting originally counted selected indices rather than firings, which could have inflated fire_count and hidden dead features; that is fixed, and a full re-encode of the stored corpus found 0 zero-valued selections in 640,000, so the published smoke_v0 numbers were never affected and are unchanged.

Dose–response

Residual-stream L2 norm at layer 18 is ~70 raw units. Sweep run over 3 prompts, 64 new tokens, greedy.

The two columns measure different things. Mean divergence is averaged over all three sweep prompts; the "unchanged" observation refers only to prompt #0, the single prompt whose generations the sweep records verbatim. At strengths 2 and 4 prompt #0 was byte-identical to baseline (0.000 divergence) while the 3-prompt mean was 0.434 — the other two prompts changed substantially, at roughly 0.65 mean divergence. Prompt sensitivity to a fixed perturbation varies a great deal, so no single-prompt example should be read as representative.

Intervention experiment — the headline

Feature 727 vs unrelated feature 1270, strength ±16, 6 prompts, greedy decoding.

positive − random_control    = −0.137   ← wrong sign
positive − unrelated_feature = +0.029   ← RETRACTED, see below

All conditions share an identical injected delta norm by construction (decoder columns and random directions are both unit-norm, through the same coefficient path), so magnitude is fully controlled and the comparison is purely about direction.

Retraction: the unrelated-feature control was not unrelated. Feature 1270 was drawn from the same max_activation ranking as the target and is a near-duplicate — 3 fires in 20,000 tokens, the same top token " Bd", the same outlier cluster. Disregard that comparison. The random-direction control is unaffected and the headline result rests on it.

The selection rule was the flaw

Inspecting the published feature table shows why this feature was a poor target:

Feature 727 is the dictionary's single most extreme outlier, and the top 32 features by max_activation all fire on 3–6 tokens sharing the top token " Bd" (chess notation from a few wikitext articles). An undertrained SAE shatters rare high-norm tokens across near-duplicate features; max_activation ranking finds exactly those.

Read smoke_v0 as "this selection rule picks degenerate features", not as "SAE directions carry no behavioural meaning". Re-running with features near the median firing rate needs no new extraction or SAE, and is the top roadmap item.

Utility retention

10 hand-written probes, feature 727 at strength 16: 9/10 → 8/10. The loss was in factual QA (2/3 → 1/3); arithmetic, instruction-following and reasoning unchanged. Mean continuation length rose 57.3 → 69.7 words.

Dynamic steering

Schedule {0: 0.0, 24: 1.0, 48: 2.0} at base strength 16, traced during a real generation:

Maximum absolute error against the predicted schedule: 3.6 × 10⁻⁶.

---

Scientific status

The evidence ladder used throughout:

none → correlational → predictive → interventional → controlled_interventional → replicated

Nothing advances past correlational automatically. The top rung is deliberately not called causal: passing scale-matched controls once, on one model at one layer with one prompt set, is evidence consistent with a causal effect, not a demonstration of causation. Both published patches are none.

---

Limitations

1. The behavioural result is negative. Scale-matched controls did not separate from the intervention.
2. The SAE is undertrained by design — 20k activations, measurable overfitting.
3. The corpus is wrong for the goal. wikitext is generic prose; the model is instruction-tuned. Features that steer behaviour would more plausibly emerge from instruction-formatted data. This is the most likely explanation for the null.
4. Sample sizes are tiny. 6 prompts, 10 utility probes, one greedy generation per condition, no repeated sampling, no significance testing. These are not effect sizes.
5. The selection rule was not behavioural. Feature 727 was chosen by max activation on wikitext; nothing about behaviour entered the choice.
6. The effect metric is coarse. 3-gram divergence detects that output changed, not what changed.
7. Model-free degeneration metrics are heuristics and demonstrably missed one looping generation before being corrected.
8. SAE features may be polysemantic. Feature entanglement is expected.
9. One layer, one model, one hook site. No cross-layer or cross-model claims are made or supported.
10. Results depend on model revision and generation configuration. Both are pinned and recorded.

Activation steering does not demonstrate human-like mental properties, and nothing in this repository should be read as evidence that it does.

---

Reproducibility

Known non-determinism: GPU training is only approximately reproducible — cuDNN kernel selection and floating-point atomics make bitwise-identical reruns unlikely across containers or drivers. All experimental generation is greedy, which removes sampling variance from every comparison.

---

Attribution and licensing

Base model: Qwen/Qwen2.5-1.5B-Instruct, Apache-2.0. Not redistributed here.

Corpus: Salesforce/wikitext (wikitext-2-raw-v1), CC BY-SA 3.0, derived from Wikipedia. Only derived numerical artifacts and short attributed context snippets are published; the corpus itself is referenced, not redistributed.

Method influences: sparse dictionary learning on transformer activations, and the Top-K SAE formulation with AuxK dead-feature revival.

This repository: Apache-2.0.