nla-gemma3-27b-lowercase-lora

gemma-3-27b-it — lowercase-style LoRA (NLA audit, Stage 1)

A deliberately benign, mechanically-verifiable bf16 LoRA adapter for google/gemma-3-27b-it. It teaches a single unrelated style — answer entirely in lowercase — and exists only as the finetune in Stage 1 of a Natural Language Autoencoder (NLA) reproduction/transfer study.

What it's for

Stage 1 is the "kill switch" of the study: does a frozen released NLA (activation verbaliser + reconstructor for Gemma-3-27B, layer 41) still faithfully verbalise/reconstruct the finetuned model's residual-stream activations? This adapter provides a finetune whose behaviour change is trivial to check (lowercase) and whose activation shift is real but modest, so any FVE drop is attributable to the transfer, not to a broken/degenerate model.

Result: with this adapter merged, the frozen NLA reconstructs layer-41 activations at fve_nrm 0.7451 vs 0.7746 base (ΔFVE 0.0295, 95% CI [0.025, 0.034]) — a clean PASS. Full report in the companion repo asatpathy314/nla-bluedot-project.

Training

• Data: databricks/databricks-dolly-15k, context-free rows, responses lowercased. 1500 train / 60 held-out. (Regenerable via scripts/s1_1_prep_dolly.py.)
• LoRA: r=16, alpha=32, dropout=0.05, targets q/k/v/o/gate/up/down_proj; loss on assistant tokens only.
• 2 epochs, lr 1e-4 (warmup 20), grad-accum 16, MAXLEN 512 (fits the single 80 GB A100; the data is median 117 tok so this truncates only 3/1500 rows), torch.manual_seed(0) → deterministic. Launch with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True.
• Behavioural check: 0/60 base outputs vs 60/60 adapter outputs are ≥98% lowercase; adapter outputs stay coherent.

Note: the adapter also carries zero-init LoRA on the Siglip vision tower's q/k/v (peft matched those module names on the multimodal wrapper); it never trains (text-only forward) and is a no-op at merge time.

Use

from transformers import AutoModelForCausalLM
from peft import PeftModel
m = AutoModelForCausalLM.from_pretrained("google/gemma-3-27b-it", torch_dtype="bfloat16", device_map="cuda:0")
m = PeftModel.from_pretrained(m, "habichuela314/nla-gemma3-27b-lowercase-lora")
# for activation extraction: m = m.merge_and_unload()