Gemma-4-26B-A4B-Preserving-Abliteration

How this model was abliterated

This model was abliterated with a custom setup built in https://github.com/CoffeeVampir3/Mojo-Gemma4-CPU/tree/ablating, running entirely on CPU with no low-rank projections anywhere. Everything you need to reproduce it is in the repo. You can run the result with this repo on CPU or just drop it into any normal inference engine — it's a regular abliterated checkpoint, nothing special about loading it.

The overall approach is close to heretic. Here's how it works, and where it diverges from heretic's current style.

What came out

• KL divergence on benign prompts: about 0.00026 (~0.026%) — basically the model's normal behavior barely moved.
• Refusals on the harmful test set: 104 down to 7.

save_abliterated: wrote checkpoints/gemma-4-26B-A4B-it-abliterated
  lambda* 2.09375: KL(full) 0.0002577923426088091
  refusals@64 7/104 (baseline 104)
  wrote checkpoints/gemma-4-26B-A4B-it-abliterated/abliteration_results.txt

Finding the direction and the per-layer schedule

First we run the model over a batch of harmful prompts and a batch of harmless ones, and record the residual activations at every layer. Before averaging anything, we winsorize each activation — clamp the outliers down to the 99.5th percentile — so a handful of crazy dimensions don't end up steering the whole thing.

Then for each layer we take the difference of means (harmful minus harmless) and normalize it. That difference vector is the refusal direction for that layer — and yeah, it's a separate direction per layer, not one shared direction.

We also work out a signal-to-noise ratio for each layer: how big that difference is relative to the activation norm, scaled so the strongest layer sits at the top. That gives us a "layer schedule" — basically how much editing each layer should get, relative to the layer that matters most.

The edit itself

Next we snapshot the baseline: the unedited model's first-token output on a set of harmless prompts. That's our reference point for measuring KL later.

The edit picks one global strength number and hands each layer its own slice of it — a layer's actual strength is that global number times its schedule weight, capped so no single layer goes overboard. At that strength, we strip the layer's refusal direction out of the three weights that write into the residual stream: o_proj, down_proj, and experts_down. The removal is norm-preserving, so we're redirecting rather than shrinking — the overall magnitudes stay put.

Everything here runs on the full weights.

To feel out the right strength, we crank it up step by step and watch the KL divergence against the baseline. Each try applies the edit, measures KL, then puts the original weights back — so the trials don't pile up on each other, and every one starts from a clean model.

Up to this point it's basically the same as heretic. The big difference: heretic derives its direction low-rank, while everything here happens on the full weights, so there's no low-rank error sneaking in.

Dialing in the strength

We treat finding the strength as a root-finding problem over a semi-fixed grid — a grid search first, then a bisection to tighten it up.

• Grid scan: sweep the strength across a fixed grid (1.0 to 3.0) and bracket the crossover — the biggest value still under budget, and the smallest one over it.
• Bisection: squeeze that bracket down toward the crossover.
• Take the biggest strength we've confirmed under budget. That's the winner.

The nice part is every check only needs the first token's output, not a whole generation, so the search stays cheap. KL basically hands us all the signal we need: edit harder and the benign-prompt drift climbs right alongside it, monotonically, so that one number is enough to steer the whole thing.

And that's it

Once we've got the crossover strength, we apply that edit to the weights and save it out. There's your abliterated model.