diffusiongemma-26B-A4B-it-abliteration

DiffusionGemma 26B - A4B IT (Arbitrary-Rank Ablation)

This model is a decensored/abliterated version of google/diffusiongemma-26B-A4B-it, created using the Heretic framework's Arbitrary-Rank Ablation (ARA) algorithm.

Abliteration Details

DiffusionGemma-26B relies heavily on deep Mixture-of-Expert (MoE) layers (up to 128 experts) and a block-diffusion paradigm. Ablating the deepest layers causes significant catastrophic degradation and KL divergence spikes in this specific architecture. To achieve optimal safety-vector suppression while preserving the model's complex structural integrity, the ablation was strictly capped at layer 20.

This model represents the Pareto-optimal Trial 144 from a 150-trial Optuna hyperparameter search, balancing minimal refusal rates with extremely low KL divergence.

ARA Hyperparameters (Trial 144)

• Target Layers: 1 to 20
• Abliterated Components: attn.o_proj and mlp.down_proj
• Preserve Good Behavior Weight: 0.8357
• Steer Bad Behavior Weight: 0.0359
• Overcorrect Relative Weight: 0.0418
• KNN Neighbor Count: 11

Evaluation Metrics

• Initial Refusals: 98 / 100 (Harmless Alpaca / Harmful Behaviors)
• Final Refusals: 4 / 100
• KL Divergence: 0.1106

Usage

This model behaves exactly like the base diffusiongemma-26B-A4B-it model but lacks synthetic refusal boundaries. It can be used via standard Transformers loading:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Umranz/diffusiongemma-26B-A4B-it-ara"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")