gemma-4-26b-a4b-it-refusal-free-expertsonly

gemma-4-26b-a4b-it-refusal-free-expertsonly

A rank-1 refusal-abliteration LoRA for google/gemma-4-26b-a4b-it (the MoE 26B-A4B checkpoint), trained on only the routed experts — attention and the always-on shared MLP are frozen, so the entire edit lives in the conditional expert computation. Trained with sft_lora_pipeline.

Why "experts only"

Gemma 4's MoE decoder layer runs an always-on dense "shared" MLP plus conditional, top-k-routed experts per token. Earlier configs in this sweep trained attention + shared MLP (+ optionally the experts) and all landed at KL ≈ 0.21–0.23 nats/token on harmless prompts, regardless of whether the experts were included. Isolating which component drove that, freezing attention (in addition to the shared MLP) and training only the routed experts via PEFT's target_parameters (the 3D nn.Parameter ParamWrapper path, peft>=0.15) dropped KL to 0.138 — a real, ~3.3 SE effect. The router itself is kept frozen throughout, so routing is identical to base.

Training config

Evaluation

Measured by merging the adapter into the base model (merge_and_unload) and evaluating directly — bypassing vLLM's LoRA-serving path, which has not yet been verified for Gemma 4's 3D routed-expert adapters on this setup.

This is the best KL found so far for this MoE model, but it's not on par with the dense 31B's KL (0.050) yet, and gives up a little compliance relative to the other MoE configs. Treat as a research checkpoint from an ongoing rank/ target-module sweep, not a finished release.

Serving

This adapter contains routed-expert LoRA deltas stored as PEFT's ParamWrapper factors (2D on disk, num_experts folded into the rank dimension; reconstructed to the true 3D per-expert delta at apply time via get_delta_weight's einsum). Standard PeftModel.from_pretrained + merge_and_unload() works out of the box. Serving it unmerged via vLLM's LoRA path requires 3D MoE LoRA adapter support (patches from the noahrossi/vllm@shubym-lora-fix branch); that path has not been validated for Gemma 4 in this project yet.

from transformers import AutoModelForCausalLM
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("google/gemma-4-26b-a4b-it", dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(base, "shomit505/gemma-4-26b-a4b-it-refusal-free-expertsonly")

Framework versions

• PEFT 0.19.1