diffusiongemma-26B-A4B-it-HERETIC-Uncensored

diffusiongemma-26B-A4B-it-HERETIC-Uncensored

This is the first abliteration of DiffusionGemma 26B A4B, produced using heretic with custom patches to support its block-diffusion architecture and MoE expert layers.

DiffusionGemma is not a standard autoregressive transformer, so this required significant engineering work that hasn't been done before for this model class.

Usage

Load using the DiffusionGemmaForBlockDiffusion class directly, not AutoModelForCausalLM:

import torch
from transformers import AutoTokenizer, DiffusionGemmaForBlockDiffusion

model_id = "edwixx/diffusiongemma-26B-A4B-it-HERETIC-Uncensored"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = DiffusionGemmaForBlockDiffusion.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="cuda"
)

messages = [{"role": "user", "content": "Your prompt here"}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True).to("cuda")

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=200, do_sample=True, temperature=0.7)

response = tokenizer.decode(out.sequences[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(response)

Results

What had to be patched

Heretic assumes standard autoregressive models. DiffusionGemma needed several custom changes:

Expert-Granular Abliteration (EGA): The MoE experts.down_proj is a batched parameter [128, 2816, 704], not a regular linear layer. Heretic skips it by default. We iterate over all 128 expert slices per layer and apply norm-preserving biprojected ablation to each one. Without this, refusals barely moved. Credit to TrevorS for the original EGA idea on Gemma 4.

Weight tying fix: The encoder and decoder share the exact same weight tensors (confirmed via data_ptr). PEFT only wraps the encoder side, so the decoder wouldn't see the LoRA delta during generation. Fixed with a context manager that temporarily merges the LoRA into the shared base weights before each generation call.

Task type: DiffusionGemma's generation mixin doesn't implement prepare_inputs_for_generation, which the default CAUSAL_LM PEFT task type requires. Switched to FEATURE_EXTRACTION.

Hidden states: DiffusionGemma's generate() doesn't support output_hidden_states. Switched to forward hooks on encoder layers to capture per-layer activations for the refusal direction PCA.

Output handling: The model returns DiffusionGemmaGenerationOutput with a .sequences attribute, not a raw tensor like standard models. Patched all heretic output handling.

Notes

This is a research release. The model will attempt to answer prompts it previously refused. 13/100 sensitive prompts still trigger refusals in testing.

---

Known issue: "own" tokens

Some token positions in generated outputs show the word "own" where other content was expected.

Example output:

"It pulls the own with own hands" (should be "It pulls the tide with gentle hands")

Initially assumed to be a diffusion denoising fallback, but kabachuha pointed out this artifact shows up in base autoregressive Gemma4 models too when weights are pushed hard (e.g. overfit LoRA training). Likely a Gemma4-level placeholder token, not specific to the diffusion architecture. A small LoRA fine-tune on clean data would probably reduce it. PRs welcome.