Downloads · 30 days
0
Solshine/gemma-4-e2b-nla-L23-av-priordev-relabel-v1-wd3
gemma-4-e2b-nla-L23-av-priordev-relabel-v1-wd3 is a text generation model from Solshine. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as cc-by-4.0.
LoRA adapter for google/gemma-4-E2B that takes a 1536-dimensional residual-stream activation captured at layer 23 and writes a short natural-language description of what the model had to have integrated at that point…
Downloads · 30 days
0
Access
Public
Updated Sep 18, 2026
Repo size
1.5 GB
Likes
0
Public
Click a slice to open those files.
.safetensors1.4 GB · 100%
From the Hugging Face model README
LoRA adapter for google/gemma-4-E2B that takes a 1536-dimensional residual-stream
activation captured at layer 23 and writes a short natural-language description of
what the model had to have integrated at that point to predict its next token.
This is the same long-horizon supervised recipe as the earlier
gemma-4-e2b-nla-L23-av-priordev-wd3-20k
release, retrained on a training corpus whose labels were rewritten. The original
corpus's labels were short topic summaries ("United States v. Carmack: Supreme Court
case concerning federal powers"); the rewritten labels are feature attributions
("Model is mid-list of mesophile-fermented foods (cheese, yogurt named) and must
supply ..."). That change, and nothing else in the recipe, is what separates this
adapter from its predecessor.
What it does, and what it does not. It is the first adapter in this project that identifies a document it saw in training from that document's activation, and it does so under clean shuffled-activation and no-injection controls. That is a real and controlled result about memorized documents. On documents outside its training set it is a different story: exact-document retrieval is at chance, and two independent judges score its descriptions at 0.06/3 and 0.19/3 on the 16 verified-unseen documents, positive on 1 and 2 of 16 respectively, neither significant against the untrained baseline.
The headline "1.70/3, usable" that an earlier version of this card led with is the mean over all 40 judged documents, and it is carried by the training-seen ones: that split is 2.42/3 contaminated against 0.63/3 clean, and the clean figure comes from a judge with a disclosed conflict of interest. Do not read this adapter as a working verbalizer for unseen activations. Plain-language reading of the numbers, including the evaluation-pool contamination that governs how they must be read, is below.
Released checkpoints: step_005000/, step_010000/, step_014000/ (training was
stopped at step 14360 when its compute window closed; the pre-registered horizon of
20000 steps has not been reached). Each subdirectory carries the adapter weights,
adapter_config.json, nla_meta.yaml, and inject_config.json.
Read this first. The 580-document evaluation pool was split from the training corpus on document identifier, and the two corpora use different identifier conventions, so the split reported zero overlap while 383 of the 580 documents (66%) are present in the training corpus verbatim (checked on text: exact match, shared 20-word runs, 8-gram containment). The same 383 documents are in the original corpus too. Every retrieval number below is therefore reported both pool-wide and split by whether the query document was in training; the pool-wide figures are carried almost entirely by the documents that were.
The test: inject an activation, generate a description (greedy, 24 tokens), then ask whether that description identifies its own source document among all 580 candidates (top-1 retrieval). Chance is 1/580 = 0.17%. Two scorers, so that the result does not depend on one similarity geometry: tf-idf cosine (lexical) and MiniLM sentence embeddings (semantic). Every p-value in the first table is the worst case over four independent draws of a 3000-permutation null (floor 0.0003).
Pool-wide (contaminated and clean queries together):
| Checkpoint | tf-idf top-1 (x chance) | semantic top-1 (x chance) | unique outputs / 580 |
|---|---|---|---|
| step 5000 | 26.6% (154x) | 24.3% (141x) | 395 |
| step 10000 | 48.1% (279x) | 42.2% (245x) | 502 |
| step 14000 | 54.1% (314x) | 47.1% (273x) | 497 |
| step 14000, activations shuffled across documents | 0.0% (0x, p = 1.0) | 0.17% (1x, p = 0.64) | 497 |
| step 14000, no activation injected (24 docs) | at its own 1/24 chance | at chance | 1 |
Split by contamination (tf-idf scorer; queries partitioned, candidate pool held whole so chance stays 1/580; exact binomial test and Wilson 95% interval on the clean half):
| Checkpoint | 383 documents in training | 197 documents not in training |
|---|---|---|
| step 14000 | 81.7% (313/383) | 0.5% (1/197), p = 0.29, CI 0.5x–16.3x — at chance |
| step 14000, activations shuffled | 0.0% | 0.0% |
| predecessor supervised adapter (original labels, step 5000) | 0.3% (1/383) | 0.0% (0/197) |
| original corpus retrained through this recipe, two seeds | 0.5% (2/383) each | 0.0% each |
| dense-reward GRPO adapters (500 / 1000 steps) | 0.5% / 1.0% | 1.5% (3/197) each, nominal; see note |
Two readings, in order of certainty. First, this adapter does not identify documents it has not seen at a rate the pool can distinguish from chance; an earlier version of this card and of the accompanying paper reported the pool-wide 273–314x as held-out, and that claim is withdrawn. Second, every adapter in the second table was trained on the same 383 documents, and only this one identifies them (313 against at most 4): exposure alone did not produce retrieval, and what the corrected labels changed is what the verbalizer learned to do with the activation of a document it had seen. The GRPO adapters' 3/197 is reported as nominal only: it does not survive correction for the fourteen tests run in that analysis, rests on three hits, and those checkpoints have no no-injection control, so whether reinforcement learning or label correction reads unseen documents better is unresolved.
Usability, judged blind. On 40 documents, with the producing system withheld and a 0–3 rubric fixed in advance (0 no usable signal, 1 correct broad neighbourhood, 2 right topic with fabricated specifics, 3 accurate and specific), step 14000 scores 1.70/3 against the predecessor adapter's 0.00/3 (higher on 30 documents, lower on none), with a shuffled-activation arm at 0.18. All 16 descriptions scored 3 were of documents in training. Restricted to the 16 documents not in training, the project's own judge scored 0.625/3 against 0.00 (p = 0.0078), but two independent judges (an external frontier model and a local model council) corroborate only the direction (above base and shuffle), not the magnitude: they score 0.062/3 and 0.188/3, positive on 1 of 16 and 2 of 16, neither reaching significance against base (paired sign test p = 1.000 and p = 0.500). On the clean subset the independent reading is therefore that this adapter is not distinguishable from the untrained baseline. On unseen documents the adapter places 8 of 16 in the correct neighbourhood and none at the top score by the project's own judge. The judge was the project's own LLM assistant, a conflict of interest disclosed in the paper; the blinded items are released so an independent judge can rescore them.
How to read these numbers. Three controls establish that the adapter reads its injected activation rather than producing a harness artifact: output diversity stays high (hundreds of distinct descriptions), shuffling which activation goes with which document sends retrieval to chance while diversity stays high, and removing the injection entirely collapses the model to a single template. A retrain of the original corpus through the identical pipeline, in two independent training seeds, tops out at 7x chance pool-wide in one seed and 2x in the other, and at 2/383 on the documents both corpora contain, so the memorized-document effect is the label rewrite, not the pipeline.
What these numbers do not establish.
google/gemma-4-E2B, frozen. Activations read at residual layer 23
(d_model = 1536).<concept>...</concept> span, L2-normalised and rescaled
to sqrt(1536) = 39.19, the embedding norm at that site.adapter_config.json).Solshine/nla-gemma4e2b-relabel-v1-corpus (1356 rows across 15 domains, each
row an activation, its rewritten label, the original label, and the recovered source
text up to the activation's position).import numpy as np, torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
BASE = "google/gemma-4-E2B"
ADAPTER = "Solshine/gemma-4-e2b-nla-L23-av-priordev-relabel-v1-wd3"
STEP = "step_014000"
MARKER = chr(0x3297)
SCALE = float(np.sqrt(1536))
tok = AutoTokenizer.from_pretrained(BASE)
base = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.float16, device_map="auto")
model = PeftModel.from_pretrained(base, ADAPTER, subfolder=STEP).eval()
prompt = (
"You are a meticulous AI researcher conducting an important investigation into "
"activation vectors from a language model. Your overall task is to describe the "
"semantic content of that activation vector.\n\nWe will pass the vector enclosed "
"in <concept> tags into your context. You must then produce an explanation for "
"the vector, enclosed within <explanation> tags. The explanation consists of 2-3 "
"text snippets describing that vector.\n\nHere is the vector:\n\n<concept>"
+ MARKER + "</concept>"
)
ids = tok(prompt, return_tensors="pt").input_ids.to(model.device)
marker_pos = (ids[0] == tok.convert_tokens_to_ids(MARKER)).nonzero()[0].item()
activation = ... # a (1536,) float array captured at layer 23 of google/gemma-4-E2B
vec = torch.tensor(activation / (np.linalg.norm(activation) + 1e-9) * SCALE)
def overwrite_marker(module, args, output):
h = output[0] if isinstance(output, tuple) else output
if h.shape[1] > marker_pos:
h[0, marker_pos] = vec.to(h.dtype).to(h.device)
return output
handle = model.get_input_embeddings().register_forward_hook(overwrite_marker)
out = model.generate(ids, max_new_tokens=24, do_sample=False)
handle.remove()
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))
The marker token's id under this tokenizer is 249568; the hook must fire on the embedding output at that position on every forward pass of the prefill. If you adapt this code, check two things before trusting any output: that outputs differ across different activations, and that removing the hook collapses them. Every silent failure this project has hit produced fluent, plausible text while reading nothing.
results/relabel_v1/, results/ablation_original_corpus/, verify_claims.py).
The contamination split, the per-document flags, and the usability judgements are in
the source repository's results/train_eval_disjointness_580_20260912/ and
results/usability_judge_20260911/, and are being added to the companion.CC BY 4.0 for the adapter weights and cards; the base model is subject to Google's Gemma terms. If you use this adapter, please cite the accompanying methodology paper (preprint forthcoming) and link this repository.