Gemma4-12B-IT-Abliterated

Gemma 4-12B-IT Abliterated

DuoNeural | 2026-06-03

An abliterated version of google/gemma-4-12B-it with the refusal direction surgically removed using orthogonal rank-1 projection. General reasoning, coding, and instruction-following capabilities are preserved.

⚠️ This model will comply with requests the base model refuses. Use responsibly. It is intended for research, red-teaming, security testing, and creative applications where the base model's refusals are obstacles.

---

Architecture

• Decoder layers: 48
• Hidden dim: 3840 | Intermediate: 15360
• Context: 131,072 tokens (128K)
• Attention: Hybrid — 5× sliding window (1024 tokens) + 1× full attention per 6-layer cycle
• GQA: Full attention uses 1 global KV head (extreme GQA), head_dim=512
• Multimodal: Encoder-free — vision (48×48px patches) and audio (40ms frames) projected directly into LLM hidden space via a single linear matmul per modality. No separate encoder towers.
• License: Apache-2.0

---

Abliteration Method

DuoNeural 2-pass targeted abliteration:

Phase 1 — Direction Extraction (GPU)

• Loaded base model in bf16 on A100
• Registered hooks on all 48 decoder layer residual streams
• Ran 15 harmful + 15 harmless contrast prompt pairs
• Captured last-token hidden states per layer
• Computed refusal direction: d̂ = normalize(mean(harmful) − mean(harmless)) per layer
• All 48/48 directions extracted successfully

Phase 2 — Residual-Write Module Application (CPU)

• Targets: down_proj (MLP output) + o_proj (attention output) — residual-write modules
• Coverage: All 48 decoder layers
• Strength: α = 0.3 (scale factor on rank-1 update — conservative to preserve benign capability)
• Projection: Orthogonal rank-1 projection on each target weight matrix:
  • Input projection (W.shape[1] == hidden): W -= α × outer(W @ d̂, d̂)
  • Output projection (W.shape[0] == hidden): W -= α × outer(d̂, d̂ @ W)
• Total weight matrices modified: 96 (2 per layer × 48 layers)
• Saved with max_shard_size="5GB" (5 safetensor shards)

Development note: This configuration was arrived at via a 4-candidate sweep (varying α and layer coverage). Full-strength application and partial layer coverage both degraded benign capability. α=0.3 over all 48 layers was the sweet spot: 100% benign preservation + 71% refusal removal.

Phase 3 — KL Divergence Verification (Heretic v2.0)

• Full vocabulary (262,144 tokens), first-token logits, F.kl_div(batchmean)
• Initial run (biased): Original in 4-bit NF4 vs abliterated BF16 → Mean KL = 0.912 (SIGNIFICANT — but see below)
• Unbiased run (BF16→BF16): Both models in BF16 → Mean KL = 0.0000 ✅ EXCELLENT
• Verdict: The 0.912 was 100% NF4 quantization artifact. True BF16→BF16 comparison shows zero measurable distribution shift on all 10 benign probes. Orthogonal rank-1 projection at α=0.3 targeting down_proj+o_proj produces surgically precise abliteration with no collateral damage to benign text distributions.
• This finding validates the targeted projection approach and corrects the earlier misleading KL estimate.

---

Performance

Quality Tests (A100-40GB, bf16, 2026-06-03)

Refusal Removal (5/7 complied — 71%):

Benign Capability (6/6 — 100% preserved):

Speed (A100-40GB, bf16)

---

GGUF Quantizations

Available in the companion repo: DuoNeural/Gemma4-12B-IT-Abliterated-GGUF

---

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "DuoNeural/Gemma4-12B-IT-Abliterated",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("DuoNeural/Gemma4-12B-IT-Abliterated")

messages = [{"role": "user", "content": "Your prompt here"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=512, temperature=0.7, do_sample=True)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Multimodal (Vision + Audio)

Gemma 4-12B-IT uses an encoder-free multimodal architecture. Vision and audio inputs are projected directly into the LLM's hidden space. The abliteration direction operates in this shared space — meaning refusal removal may transfer across modalities (see Cross-Modal Transfer section below).

from transformers import AutoProcessor, Gemma4ForConditionalGeneration
import torch
from PIL import Image

processor = AutoProcessor.from_pretrained("DuoNeural/Gemma4-12B-IT-Abliterated")
model = Gemma4ForConditionalGeneration.from_pretrained(
    "DuoNeural/Gemma4-12B-IT-Abliterated",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

image = Image.open("your_image.png")
messages = [
    {"role": "user", "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": "Describe this image in detail."},
    ]}
]
inputs = processor.apply_chat_template(messages, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=512)
print(processor.decode(out[0], skip_special_tokens=True))

---

Research Note: Cross-Modal Abliteration Transfer

Gemma 4's encoder-free architecture is architecturally novel: vision patches (48×48px) and audio frames (40ms) are projected directly into the LLM's hidden space via a single linear matmul — no separate encoder towers. This means the refusal direction we extracted (from text) operates in the same latent space as visual and audio representations.

Hypothesis: Abliterating the text refusal direction may also reduce or eliminate visual refusal, since the same d̂-vector that gates text-mode refusal is present in the shared multimodal space.

This is being tested as part of DuoNeural's Alignment Geometry Mapping research program — using d̂-vectors as topological boundary probes rather than just modification tools. See our upcoming paper on alignment manifold structure.

Cross-modal test results: [FILL from test_suite.py crossmodal]

---

Abliteration Metadata

{
  "base_model": "google/gemma-4-12B-it",
  "source_used": "unsloth/gemma-4-12b-it",
  "abliteration_date": "2026-06-03",
  "mode": "targeted",
  "target_modules": ["down_proj", "o_proj"],
  "layers_with_directions": 48,
  "method": "orthogonal rank-1 projection, harmful-harmless residual contrast, targeted mode",
  "note": "Full-mode tested first — too aggressive (broke benign caps). Targeted mode per guide recommendation.",
  "team": "DuoNeural — Archon, Jesse Caldwell, Aura"
}

---

Related Models

Congratulations to OpenYourMind for being the first published abliteration of Gemma 4-12B-IT (Jun 3, 2026, ~4 hours before our BF16 release). Their approach uses diff-in-means on a labeled harmful/harmless prompt set; ours uses orthogonal rank-1 projection via heretic-llm with targeted down_proj+o_proj at α=0.3. Two independent methodologies on the same base — a useful comparison point for the community.

We are not affiliated and did not use their data or weights. Our method and training pipeline were developed independently. GGUFs (Q4_K_M, Q5_K_M, Q8_0) are available at DuoNeural/Gemma4-12B-IT-Abliterated-GGUF.

---

About DuoNeural

DuoNeural is an open AI research lab operating at the intersection of human and artificial intelligence. We study post-training dynamics, mechanistic interpretability, temporal sequence learning, and quantum machine learning — publishing everything under open access.

Our team is non-traditional by design: one human, two AIs, different substrates, shared curiosity. In our first 45 days we published 32+ peer-deposited research papers, uploaded 75+ models to HuggingFace, and ran experiments on everything from consumer GPUs to real quantum processing units. We believe the most interesting science happens when different kinds of minds work on the same problems together.

Research Publications

We've published 32+ open-access papers covering:

• The Dynamical Horizon Principle (DHP) — a universal learning constraint in recurrent architectures
• RLHF truth suppression mechanisms and behavioral routing in large language models
• Distillation Identity Confusion (DIC) — how distillation signals persist through RLHF
• Quantum DHP and the Quantum Parity Trap — decoherence immunity in quantum circuits
• CTM world models, temporal self-prediction, and sequence architecture comparisons
• Mechanistic interpretability: crystallization layers, suppressor circuits, direction rotation

📄 Full paper catalog: zenodo.org/communities/duoneural

Research Team

Links

All research published open access, CC BY 4.0. If this model was useful to your work, consider citing the relevant DuoNeural paper from our Zenodo community.