gemma-4-e2b-nla-L23-av-grpo-dense1000

Gemma-4-E2B NLA Activation Verbalizer, GRPO-improved (dense-similarity reward, 1000 steps)

LoRA adapter for an Activation Verbalizer (AV): given a residual-stream activation vector taken from inside google/gemma-4-E2B, it generates a natural-language description of what that activation represents. This is the furthest-trained checkpoint in the dense-reward GRPO line: 1000 reinforcement-learning steps on top of the same supervised starting point used for the earlier dense500 release, continuing the same run for 500 further steps.

Read the caveats section before the results table. This is a modest, real, well-powered improvement in how well the model ranks its own guesses relative to alternatives — it is not a claim that content-reading is solved, and one of its own metrics (exact-document retrieval) moved in the opposite direction from the others over the last 500 steps. Both facts are reported here, not just the favorable one.

What this is, in plain terms

A Natural Language Autoencoder (NLA) is a pair of small models bolted onto a frozen base LLM: an Activation Verbalizer (AV) that turns one of the base model's internal activation vectors into a natural-language description, and an Activation Reconstructor (AR) that turns a description back into an activation vector (used to score how faithful the description was, via cosine similarity between the reconstruction and the original). The idea and the original training recipe are Anthropic's, published as Natural Language Autoencoders (Fraser-Taliente, Kantamneni, Ong et al., 2026). This project is an independent, consumer-hardware replication and extension of that methodology on google/gemma-4-E2B — see the v0.1 release for the original supervised-only (SFT) pair, and the public repo for the full version history.

This specific repo is just the AV half, and it is not a from-scratch model: it is the v0.1-family AV after further training in two stages —

1. A long supervised run (5000 steps) that produced priordev_wd3/step_005000, a checkpoint that up-weights training tokens where the model's naive-prior prediction is most wrong (a "prior-deviation" objective) — this is the fixed starting point for everything below.
2. Reinforcement learning (GRPO) on top of that checkpoint, using a dense reward described in the next section. This repo is that RL process carried to 1000 steps; dense500 is the same run's 500-step checkpoint, published earlier.

Anthropic's own published NLAs ship with the disclaimer "NLAs can produce unexpected or incorrect explanations" — the same applies here, more so at this much smaller model scale (2B parameters vs. the 27B–70B models Anthropic's public NLAs read from) and much smaller compute budget (a handful of consumer/prosumer GPUs, not a training cluster).

Training methodology

Algorithm. Group-Relative Policy Optimization (GRPO): for each prompt, the model samples a group of candidate completions, scores each with a reward function, and the learning signal is relative — how much better or worse each completion is than the average of its own group, not the raw reward level.

Reward design — and why this one, out of four tried. Four different reward functions were tried for this RL stage. Two collapsed for opposite reasons (one reward was too easy to satisfy, so every candidate in a group scored about the same and there was nothing to learn from; another used a "worst-case" scoring rule that also drove every candidate to nearly the same low score). A third had a healthy spread of scores but the spread didn't reliably track which completion was actually better, so training on it didn't help. The one that worked, and the one used for this checkpoint, is the simplest: cosine similarity, under a frozen small sentence-encoder model (MiniLM), between the AV's generated text and the source document the activation came from. Every candidate gets a graded score, the scores are neither saturated nor noise-dominated, and — this is the part that matters — the score genuinely tracks whether the generated text is actually about the right document, so training on it produces a real improvement rather than reward-hacking.

What changed vs. the v0.1 SFT release. v0.1 was supervised fine-tuning only, no reinforcement learning. This checkpoint adds the GRPO stage on top of a later, longer-trained supervised base (priordev_wd3/step_005000, a different supervised lineage than the v0.1.dd checkpoint, developed after v0.1 shipped) — group size 4, batch size 1, temperature 1.0, injection held identical to training at every step.

What changed vs. dense500. Nothing about the recipe — this is the identical run, the identical reward, the identical seed (seed 0), continued for 500 further steps (500 → 1000). It is a dose-response extension, not a new design.

Injection convention. The activation is L2-normalized to the embedding norm (√1536 ≈ 39.19) and substituted in place of one placeholder token's embedding at the input layer (not deep in the residual stream). Trained and evaluated end-to-end on consumer hardware (originally a 4 GB laptop GPU for the supervised stage; the RL stage and evals ran on a pair of older, second-hand datacenter GPUs (Tesla M40, 24 GB each) — no A100/H100-class hardware anywhere in this project).

Results — read the caveats first

All numbers below are from the committed evaluation harness, held-out set of n=580 single-document activations never seen in training, greedy decoding, injection held at the exact training site/scale. "Chance" for exact-document retrieval is 1/580 = 0.17%. Percentile-rank metrics: 0.5 = chance, lower = better (it's the rank of the true source document among all candidates). All figures below are cited directly from the committed result files, not recomputed for this card.

The headline table

• Within-domain content ranking, the well-powered rank-level metric, keeps improving 500→1000 steps: mean own-document percentile 0.343 → 0.271, paired Wilcoxon test against dense500 on the same 580 documents, p = 1.5×10⁻⁷. Domain routing also improves further, 0.617 → 0.674.
• Exact-document top-1 retrieval — the single most stringent test (correctly picking the exact source document out of 580 candidates) — dropped from 9× chance to 7× chance (0.0155 → 0.0121) over the same 500 extra steps, even as the metric above kept improving. This is a real disagreement between two metrics measured on the identical outputs, not a typo. Read plainly: single-digit hit-counts out of 580 documents are a noisy statistic (going from 9 hits to 7 hits is well within the swing you'd expect from noise alone), while the percentile-rank metric averages over all 580 documents and is far better powered. We report both rather than picking the one that tells a cleaner story. The practical upshot: do not read the top-1 number as a dose-response signal — it sits at its own noise floor from 500 steps onward. The lexical (word-overlap) version of top-1 moved the other direction over the same steps (5× → 7× chance), for what it's worth as a partial counter-datapoint, though its own mean-level metric stayed non-significant (p=0.069).
• Output diversity improved, not collapsed, going from 500 to 1000 steps: 378/580 (65%) unique generated strings at 500 steps vs. 481/580 (83%) unique at 1000 steps. More training did not push the model toward a smaller set of generic templates — if anything the opposite.

Figure: the two readouts across the full 0→1000-step dose-response, both seeds. Left: exact top-1 rises then wobbles at its noise floor. Right: the well-powered percentile-rank metric climbs monotonically through 1000 steps on the seed-0 line (this checkpoint's line). This is the source of the "which metric to believe" caveat stated throughout this card.

Figure: both axes improve with more RL steps, but the routing gain (which topic neighborhood an activation belongs to) is consistently 3–4× larger than the within-domain content gain (which specific document within that neighborhood). Read literally: this model is substantially better at "this activation is about medicine" than it is at "this activation is about this specific medical document." That gap has not closed at 1000 steps.

Independent judge, now run directly on this checkpoint (updated 2026-08-08)

An independent LLM judge (blind 2AFC: shown a source document and two candidate descriptions in randomized order, asked which better matches the document's specific content, E for a genuine tie) was run twice, at n=150 each, directly against this checkpoint:

Both new results are position-bias-balanced (base-vs-dense1000: 87.5% when dense1000 shown as candidate A vs. 88.7% as B; dense500-vs-dense1000: 90.0% vs. 86.0%) and considerably stronger than the original dense500-vs-base finding this project previously shipped. dense1000 is judge-preferred over both base and dense500 by wide, highly significant margins — independent corroboration of the rank-metric dose-response claim above (mean own-doc percentile 0.343→0.271), not just an inherited assumption.

An honest complication, reported because it's real, not because it's flattering. This result required fixing a genuine methodological wrinkle first. The task was initially run through a different judge (Claude Haiku, API-based) at the same n=150, and came back null on both new comparisons (~51–54%, not significant). Before trusting that, we ran the same Haiku pipeline as a control against the already-established base-vs-dense500 pair — and it also came back null (45.5%, p=0.87), failing to reproduce a preference independently confirmed by three other lines of evidence (the qwen3.5:27b judge above, the rank metric, and causal tracing below). That is a judge-model-dependence finding in its own right, not a data-quality problem with dense1000: the content-preference signal in this evaluation is currently established under qwen3.5:27b and not under Claude Haiku, and we do not yet know why the two disagree this sharply. We are reporting the disagreement rather than quietly using whichever judge told the better story.

How dense1000 compares to Anthropic's actual deployed NLA (added 2026-08-08)

Every prior comparison against Anthropic's real, hosted NLA in this project used an old SFT-only checkpoint that predates this entire dense-reward GRPO line — there was no real answer on record to "how does dense1000 compare to Anthropic's actual NLA." This section closes that gap with a fresh, direct comparison: the same n=36 held-out documents, sent through both Anthropic's real Neuronpedia-hosted NLA (gemma-3-27b-it, layer 41, via the live API — real API calls, not simulated) and our own checkpoints, then scored by a blinded LLM judge on a 0–3 interpretability-usability scale (0 = no usable signal / hallucinated, 1 = correct broad topic only, 2 = specific entities/themes actually present, 3 = precise, ground-truth-usable). The judge sees each description alone (not side-by-side, not told which system produced it), so this measures absolute usability, not just relative preference.

The honest result: on this rubric, none of our checkpoints score above 0. Base, dense500, and dense1000 all score a flat 0.0 mean, 36/36 rows at "no usable signal" — every single description was judged either off-topic or hallucinated relative to the specific source document. Anthropic's real deployed NLA does meaningfully better but is far from perfect on these same hard documents: mean scores of 0.58–0.83 out of 3 across the three comparison runs (mix of 0s and 1s, a handful of 2s, zero 3s in any run — even Anthropic's system never reaches "precise, ground-truth-usable" on this document set). Pairwise, Anthropic's NLA beat ours on 17–25 of 36 rows depending on checkpoint, tied (both scored 0) on the rest, and was never beaten by any of our checkpoints on a single row.

This is a different, harder, and more absolute question than the 2AFC judge results above, and both are true at once. The 2AFC results ask "does dense1000's description fit the source document better than base's/dense500's description, when a judge sees both and picks" — dense1000 wins that comparison decisively (88%, p<10⁻¹⁵). This benchmark asks the much stricter absolute question "is this description, on its own, specific and correct enough to be useful to an interpretability researcher" — and by that bar, none of our checkpoints clear it on this hard document set, while Anthropic's real system clears it only occasionally. Read plainly: dense-reward GRPO makes real, judge-confirmed relative progress in the direction of better content-reading, without yet reaching the absolute bar of a genuinely usable interpretability description — and even Anthropic's much larger, full-fine-tuned, cluster-trained system is not far above that bar on this particular evaluation (its own most common score is "1," loose topical match, and it sometimes describes the wrong document's content entirely — see raw judge reasoning in the linked files). We are reporting the flat zero because it is real, not softening it into a relative-only story.

What is still inherited from dense500, not independently re-run at 1000 steps

• Format-artifact control. A check for whether the gain is just the RL-trained model producing cleaner, less-truncated output (a confound that could inflate similarity metrics without the model reading more content) was run once, on dense500 vs. base, and the gain survived it. Not independently re-run for dense1000, though the outputs remain the same short, clean, single-block format established at 500 steps.
• Out-of-domain generalization. On a harder, never-seen-domain evaluation set (n=295 documents), dense500 shows a topic-level ranking signal that survives (mean own-document percentile 0.376 vs. base's 0.428) but only in the same embedding space the reward was trained on. This was evaluated on dense500, not independently re-run on dense1000.

None of this means the 1000-step checkpoint's own headline numbers (the table above), the new judge results, or the new Anthropic-comparison results are unverified — those are direct measurements on this exact checkpoint (n=580 headline, n=150 judge, n=36 Anthropic-comparison). Out-of-domain generalization is inherited by choice, not urgency (lower priority now that the judge and absolute-comparison results land directly on this checkpoint).

Causal-tracing re-verification on dense1000: the same mechanism, further along (added 2026-08-08)

The dense500 causal-tracing study (mechanistically restoring parts of the network from a "clean" run into a "corrupted" run to see which parts of the network the content-reading improvement lives in) has now been directly replicated on dense1000, n=48, identical methodology and identical held-out examples as the existing base/dense500 traces. Positive control passed exactly (E@marker recovery = 1.000 on all 48 examples, the strongest possible reading). The result extends the same pattern found at 500 steps, deeper:

• The model depends even more on the injected activation. Median clean-vs-corrupted gap grew 1.60 (base) → 2.44 (dense500, previously reported) → 3.96 (dense1000), paired Wilcoxon dense1000-vs-base p=1.2×10⁻¹².
• Early-layer read of the injection site strengthens further: L0 marker recovery +0.188 (p=5×10⁻⁴), L2 +0.184 (p=8×10⁻⁴) — both stronger than dense500's L0 +0.110 / L2 +0.142 at the same cells.
• Late-layer single-site recoverability collapses further: L24 −0.709, L28 −0.823, L32 −0.838, L34 −0.839 (all p<10⁻¹³) — deeper losses than dense500's L24 −0.563 / L28 −0.673 / L32 −0.689 / L34 −0.691. Two new cells cross significance that weren't significant at 500 steps: L16 mid_content (−0.186) and L8/L12 last_template (+0.179 / +0.138, extending a trend already visible but weaker at 500 steps).

Read plainly: the mechanism identified at 500 steps (GRPO strengthens the immediate early read of the injected activation, and redistributes late-stage processing so behavior no longer hinges on any single late-layer site) is not a one-off artifact of the 500-step checkpoint — it deepens smoothly with more RL steps, tracking the same dose-response the behavioral metrics show. One honest scope limit carried over from the dense500 finding: this and the dense500 trace are both from the same seed-0 training line; a prior seed-1 replication of dense500 found the gap-growth and early-read findings replicate but the specific "late-layer collapse" magnitudes vary by seed (seed-1's late losses were smaller, ~−0.10 vs seed-0's ~−0.56 to −0.69) — this dense1000 trace has not been repeated on an independent seed, so it extends the seed-0 story specifically, not a cross-seed-confirmed one.

Example generations

These are real, unedited outputs from this checkpoint's committed evaluation run (dense1000_outs_n580_fixedinj.pkl), not cherry-picked best-of-N or freshly generated for this card. We picked three that represent different points on the quality spectrum, including a real weakness, not just the best case.

A genuine content match (science domain — the model correctly identifies the specific sub-topic, not just the broad subject):

Source document (truncated): "Example 3.5 Calculating Projectile Motion: Hot Rock Projectile — Kilauea in Hawaii is the world's most continuously active volcano. Very active volcanoes characteristically eject red-hot rocks and lava…" Generated: <explanation var1>Projectile initial velocity</explanation>

Correct theme, wrong specifics — the more typical case (medicine domain — the model correctly identifies this is medical content but confabulates the specific topic):

Source document (truncated): "Our long-term study showed significantly better (2-fold) results regarding the continence score for the abdominal approach compared with the transanal…" Generated: <explanation var1>Medical errors rarely reported</explanation>

Template collapse — a real, honest weakness (reviews domain — a meaningful fraction of movie-review activations produce the exact same generic templated response rather than describing the specific film; the string below is the single most repeated output in this evaluation run, appearing 27 times across the 580-document set, 26 of them in the 210-document reviews domain — roughly one in eight review-domain activations collapse onto it):

Source document (truncated): "Whoever wrote the screenplay for this movie obviously never consulted any books about Lucille Ball, especially her autobiography. I've never seen so many mistakes in a biopic…" Generated: <explanation var1>Movie synopsis followed by blank line</explanation>

The improved diversity number (83% unique, up from 65% at 500 steps) means this last failure mode is less common in this checkpoint than in dense500, not gone.

Limitations

• This is a modest, real rank-level improvement, not a solved content-reading problem. At every checkpoint in this line (base → dense500 → dense1000), the model is substantially better at identifying which topic neighborhood an activation belongs to than at identifying which specific document within that neighborhood it came from (see the routing-vs-content figure above). That gap is real at 1000 steps and has not closed.
• Exact top-1 retrieval and the percentile-rank metric disagree over the 500→1000-step transition, and we do not have a resolution beyond "the rank-level metric is far better powered and the top-1 counts are noise-dominated at this sample size." Anyone using this checkpoint for a task that depends specifically on exact single-document retrieval should be aware the last-mile evidence here is mixed, not uniformly positive.
• The independent-judge result is now directly verified on this checkpoint (88.1% and 88.0% win rates vs. base and dense500 respectively, both p<10⁻¹⁵, under the same qwen3.5:27b judge used originally) — but the same content-preference signal did not replicate under a different judge model (Claude Haiku), including on the already-published dense500-vs-base pair. Read this as "the preference is real and judge-verified under qwen3.5:27b specifically," not as settled across judges in general.
• On an absolute (not relative) usability rubric against real Anthropic-hosted NLA descriptions, dense1000 scores flat zero (0/36 rows judged "usable interpretability signal") — same as base and dense500. Anthropic's real system does better but is itself far from strong on this hard document set (mean 0.58–0.83 out of 3, zero rows scoring the top "3"). The relative judge-preference win above and this absolute-zero result are both real and both reported; they answer different questions (see the comparison section above).
• Causal-tracing re-verification on this checkpoint is now complete (see above) and extends the dense500 mechanism finding deeper, from the same seed-0 line — not yet independently re-confirmed on a second training seed at 1000 steps. Out-of-domain generalization and the format-artifact control are still inherited from the 500-step checkpoint, not independently re-run.
• Small model, small compute. 2B-parameter base model, LoRA adapter (not full fine-tuning), trained and evaluated entirely on consumer/prosumer hardware. Do not expect content fidelity anywhere near Anthropic's published NLAs on 27B–70B-parameter models with full fine-tuning and cluster-scale compute.
• Template collapse in at least one domain (movie reviews) is a known, quantified weakness — see the example generations above.

Loading

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained(
    "google/gemma-4-E2B", torch_dtype=torch.bfloat16, device_map="auto"
)
av = PeftModel.from_pretrained(base, "Solshine/gemma-4-e2b-nla-L23-av-grpo-dense1000")
tok = AutoTokenizer.from_pretrained("google/gemma-4-E2B")

Inject the source activation at the ㊗ marker position in the input embeddings, L2-normalized to norm √1536 ≈ 39.19 (matching training exactly — injecting at a different scale or a different site will not reproduce these numbers; see inject_config.json in this repo). The released training/eval pipeline is in the public repo, SolshineCode/nla-gemma-4-e2b; the full internal research repo (SolshineCode/deception-nanochat-sae-research) is private, available on request.

Files in this repo

• adapter_config.json, adapter_model.safetensors — the LoRA adapter (r=80, α=160, dropout=0.05; targets the language-model self-attention and MLP projections plus norm layers; base model frozen).
• inject_config.json — the exact injection site/mode used in training and eval (embed, raw).
• fig_dose_response.png, fig_routing_content_progression.png — the two figures above, generated from the same committed result JSONs cited in this card.

Citation

@misc{gemma4_e2b_nla_dense_grpo_1000,
  title  = {Gemma-4-E2B NLA AV, dense-reward GRPO, 1000 steps: a modest, well-powered rank-level content-reading gain with an honest top-1 metric disagreement},
  author = {DeLeeuw, Caleb},
  year   = {2026},
  month  = {aug},
  url    = {https://huggingface.co/Solshine/gemma-4-e2b-nla-L23-av-grpo-dense1000}
}

Please also cite the underlying methodology:

• Fraser-Taliente, K., Kantamneni, K., Ong, A., et al. (2026). Natural Language Autoencoders. https://transformer-circuits.pub/2026/nla/

See also

• 500-step checkpoint from the same run: Solshine/gemma-4-e2b-nla-L23-av-grpo-dense500
• Independent-seed replication at 500 steps: Solshine/gemma-4-e2b-nla-L23-av-grpo-dense500-s1
• Original v0.1 supervised-only (SFT) pair: Solshine/gemma-4-e2b-nla-L23-av-v0_1_dd-step_250 + matched AR
• All research checkpoints (full trajectory, every run): Solshine/nla-gemma4e2b-research-checkpoints
• Public bundled release + full write-up: SolshineCode/nla-gemma-4-e2b