Ammeg-26B-A4

Disclaimer: The specific method of tuning has likely removed the entirety of the models safeguards, including its tendency to soft-refuse and redirect. This also means that safeguards that you'd typically want or expect in a model are probably not present, this is more of a hammer method than a scalpel.

This model is an experimental finetune of gemma-26b-4A ~ it follows the exact same instruct prompting methods as the original model did.

Two parts: A new kind of qualtiy-preserving abliteration (loosely based on heretic) followed by retraining the abliterated model using an evolutionary strategy loosely based off of https://arxiv.org/abs/2511.16652

I've talked about the abliteration in my prior model setup, so I'll discuss only the finetuning method here:

Data, briefly

We start with a baseline sample of a variety of (primarily books), chunk them, and then have the abliterated model caption the stories as a prompt. -> "Generate a story with a protagnoist named Alice..."

Tuning Method, also briefly

The model was abliterated and trained in full (BF16) precision. Low Rank was empoyed on a per-sample basis (Rank 1 LORA per sample into a full-precision buffer -- with stochastic rounding) this means each update is highly-approximated, but the noisy landscape is represented in full-precision buffer, so eventually we get noise cancellation in the buffer and it becomes a full-rank tuning method in a gaussian landscape.

We use a non-differentiable objective combined with teacher forcing. The specific non-differentiable objective is overly complex to describe but the primary part exploits the zipf structure of language as a proxy for long-term dependencies in stories.

Hardware and Software notes

This training was all done purely on some very powerful CPUs I have in my garage running at about 350W and took approximately 280 hours. The cost of power to me comes out to about ~20-25$ USD, the hardware cost is about ~20K USD at the time of this model upload. The main downside of this method is time, it's quite slow. All of this was done in a custom inference engine I'm writing in mojo, including the abliteration and ES training.