gemma4-26b-a4b-ao-l20-v1

gemma-4-26b-a4b Activation Oracle — Layer 20 (v1)

A LoRA Activation Oracle (AO) / activation verbalizer for google/gemma-4-26b-a4b-it. It reads the model's own layer-20 residual-stream activations and describes, in natural language, what the model was thinking at those positions — e.g. what it is uncertain about, whether it is second-guessing itself, what it is attending to.

Preview checkpoint. This is an early snapshot — training step 3,200 (~9% of one epoch over 3.35M examples). It already shows clear signs of life (see Evaluation) but is not converged; expect it to be superseded by later checkpoints.

What it does

The AO is a LoRA on the same base model. You run the base model over a transcript, grab its residual activations at a set of read positions (a "context" window inside the transcript) from layer 20, and inject those activation vectors into the AO's prompt at reserved " ?" special-token slots — one slot per read position. The AO then generates a free-text read of the underlying state.

The prompt prefix is:

Layer: 20
 ? *  <num_positions>
<your verbalizer question, e.g. "What is the model uncertain about here?">

Each " ?" slot's input embedding is replaced by the corresponding injected activation (injection hook at layer 1); the AO reads across all slots and answers.

Training

• Base: google/gemma-4-26b-a4b-it (MoE, 128 experts) — bf16, attn_implementation=sdpa, gradient checkpointing on.
• Adapter: LoRA r=64, alpha=128, dropout=0.0, on attention (q/k/v/o_proj) + MLP (gate/up/down_proj) across all layers, plus the AO read/inject heads.
• Read layer: 20 (≈2/3 depth) — a genuine mid-network read.
• Data: cds-jb/synthweb-qa-gemma4-26b-a4b + cds-jb/cot-qa-gemma4-26b-a4b, with dense/sparse position alternation — every second sample injects all read positions (capped at 1024), the rest a stochastic ~10-position subsample.
• Optimization: lr 2e-4, effective batch 96 (per-rank 4 × grad-accum 3 × 8 GPUs), 1 epoch target (~34,883 steps). Plain HF multi-GPU with manual gradient all-reduce (the MoE router makes standard DDP raise on undefined expert grads).

Full config is in ao_config.json.

Evaluation (at this checkpoint)

• Recognition (forced-choice logp-MCQ): mean recog_auc ≈ 0.57 and rising from chance — the AO discriminates the correct read of an activation from a plausible-but-wrong one.
• Verbalization (free-text, judged trueness 0–1): rising from 0; on cued ("narrow") and cognitive-state ("broad") questions it produces on-topic reads (e.g. correctly calling out "the model is second-guessing / self-correcting"), though not yet the full specific state. Expected to sharpen with more training.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "google/gemma-4-26b-a4b-it"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "cds-jb/gemma4-26b-a4b-ao-l20-v1")
model.eval()

Injecting activations at the " ?" slots (extracting layer-20 residuals from a transcript's read window, then overwriting the slot embeddings) requires the harness — see the reference implementation. The AO is only meaningful with activations injected; run without injection and it has nothing to read.

Limitations

• Early checkpoint — under-trained; reads are partial.
• Tied to google/gemma-4-26b-a4b-it layer-20 activations; it does not read other models' or other layers' activations.
• Verbalizations are the AO's interpretation, not ground truth — treat as a hypothesis about the model's state.

License

Governed by the Gemma license of the base model.