gemma-emotion-vectors

Gemma Emotion Vectors

Emotion direction vectors from Gemma 2, released for two studies.

• Replication study (repository root). 171 emotion direction vectors extracted from the base model google/gemma-2-27b, plus a recipe to regenerate the logit-lens ingredients. Companion artifact to a protocol-matched replication of Anthropic's emotion-vector study (Sofroniew et al. 2026) on a base model. Paper: Replicating the Geometry of Emotion Representations in a Base Open-Weights Model, arXiv:2609.22208.

• Six-model grid (gemma_grid/). The same pipeline applied to Gemma 2 at three sizes (2B, 9B, 27B), each in its base and instruction-tuned form. For each model: 171 emotion vectors at every layer, neutral-dialogue activations and cleanup bases, per-story activation slices, bootstrap draws and embedding-table vectors, plus probe means and colon-token activations. Base 27B reuses the replication's vectors above. Paper: The Geometry of Emotion Representations Across Scale, Depth, and Instruction Tuning, arXiv:2610.35411. See gemma_grid/README.md. Its code, results and checksums are in the code repo's gemma_grid/ folder.

• Code: https://github.com/adamhollowell/emotion_probes

• Corpus the vectors were extracted from: 205,200 Claude-Sonnet-4.5-generated stories, released as a generation recipe (the prompt + topic/emotion design) in the code repo and the replication paper's Appendix A — not redistributed as a file. See the code repo's CLAUDE_DATA_NOTICE.md.

Files

The files below are the replication study's, at the repository root. The grid's files are listed in gemma_grid/NOTICE and in the code repo's gemma_grid/SHA256SUMS.txt.

• emotion_vectors_171.pt — the emotion vectors. A torch.save dict:
  • emotions — list of 171 emotion labels
  • vectors — [171, 46, 4608] float32, confound-cleaned (the canonical set)
  • vectors_raw — [171, 46, 4608] float32, before cleanup
  • n_layers = 46, d_model = 4608, variant
• embedding_null_vectors.pt — the embedding-layer baseline (§6.6): 171 emotion vectors built from Gemma's input embedding table (embed_tokens.weight) with no forward pass — a bag-of-embeddings null. Same dict schema, shape [171, 1, 4608] (vectors cleaned + vectors_raw, layer = "embed_tokens", provenance). Regenerate with extract_embedding_null.py (code repo).
• export_lens_ingredients.py — regenerates lens_ingredients.pt (the unembedding W_U + final-RMSNorm weight + vocab) from the base model. Not shipped as a file because it is essentially Gemma-2-27B's own weights; it is fully re-derivable from the base model (see below).
• pertoken_stripped.tar — the text-stripped per-token projection artifact behind Figure 1 / §8 (one .npz per winning document: token ids, per-token projection values, random-direction projections, norms, doc/source labels, offsets — every readable-text field removed). Hosted here because it is 549 MB, too large for the GitHub code repo. Provenance / license — read this: this file is not a Gemma derivative and is not covered by the repo-level Gemma license below. It is derived from the held-out evaluation corpus (four third-party datasets, §3.6), text-stripped so that no third-party text is redistributed; downstream use is subject to those upstream sources' terms. The code repo's RUN_MANIFEST.json records its full provenance and the strip_pertoken_text.py producer.

Checksums (SHA-256). For provenance, the shipped binaries are:

• emotion_vectors_171.pt — 6de2e293d0e642bd164111e3be24917d7460bd4810fffa6c710cd1a97235971b (100T basis; this sha256 is the authoritative basis identifier and is the value every analysis script sha-gates against. The file's internal variant field carries a legacy neutral_HA label from the extraction tooling — ignore it; the sha is what governs.)
• embedding_null_vectors.pt — 476230a0406c462e4769193dbe2fdc27846c2e5a92ebc0a5190efdf45874d5ea
• pertoken_stripped.tar — a30ce7894ce09c171bf162787a12d7a9559114d108f1667fe3ca2ac51439f2d6 (text-stripped per-token artifact; held-out-corpus-derived — see the provenance note above, not Gemma-licensed)

from huggingface_hub import hf_hub_download
import torch
d = torch.load(hf_hub_download("adamhollowell/gemma-emotion-vectors", "emotion_vectors_171.pt"),
               map_location="cpu", weights_only=False)
v_afraid_L31 = d["vectors"][d["emotions"].index("afraid"), 31]   # a 4608-d direction

How the vectors were extracted (replication study)

Faithful to the original study's method (parameters inherited, not tuned on Gemma):

• Base google/gemma-2-27b via TransformerLens 3.3.0 TransformerBridge, raw-weights regime (tl_compatibility_mode=False), bfloat16, on a single 80 GB H100.
• Residual stream resid_post at all 46 layers, mean-pooled over token positions from the 50th on.
• Each emotion vector = mean(that emotion's stories) − grand mean over all 171 emotions.
• Confound cleanup: per-layer, project out the top neutral-activation PCs explaining ≥50% of variance (procedure + threshold inherited from Anthropic; neutral set = the 500 neutral dialogues).
• Analysis layer of record: L31 (~⅔ depth), the original's measurement layer.

Geometry and activation analyses run from emotion_vectors_171.pt on CPU — no GPU needed.

Logit-lens ingredients ship as a recipe, not a file

lens_ingredients.pt is ~4.4 GB and is, to within rounding, Gemma-2-27B's unembedding matrix W_U (1.18 B params) — a verbatim slab of Gemma's weights. Rather than redistribute Gemma weights, run:

python export_lens_ingredients.py --out lens_ingredients.pt

It reads only the embedding + final-norm tensors from the base model's safetensors (~9 GB, CPU-ok; requires Hugging Face Gemma access) and reproduces the original ingredients bit-for-bit.

Caveat — Claude-rendered stimuli

These vectors describe how base Gemma represents emotion as Claude Sonnet 4.5 renders it in short fiction (the extraction corpus), not emotion in the wild. That provenance is in the causal chain of every geometry, clustering, and human-norm-alignment result. It does not weaken the findings, but it should be disclosed if you build on the vectors.

Gemma provenance and redistribution (Gemma Terms of Use §3.1)

Both emotion_vectors_171.pt and embedding_null_vectors.pt are Gemma Model Derivatives as defined in the Gemma Terms of Use §1.1(e): they are "created by transfer of patterns of the weights, parameters, [and] operations… of Gemma" — difference-of-means over google/gemma-2-27b residual activations, and (for the null) a direct transform of its embedding table. They are therefore distributed under and subject to the Gemma Terms of Use, and this redistribution meets the §3.1 conditions:

• Notice file (§3.1.4): NOTICE accompanies this distribution and carries the required text, "Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms."
• Copy of the Agreement (§3.1.2): a copy of the Gemma Terms is included as GEMMA_TERMS.md; the canonical version is at ai.google.dev/gemma/terms.
• Use restrictions pass through (§3.1.1): your use of these artifacts is subject to the Gemma use restrictions in §3.2, which incorporate the Gemma Prohibited Use Policy. By using or further distributing these files you agree to those restrictions and must pass them to anyone you distribute to.
• Modified-files notice (§3.1.3): these are newly created derivative artifacts, not edited copies of Gemma's own files; their derivative provenance is stated here, in NOTICE, and in each file's provenance field.

Google claims no rights in Gemma Outputs (§3.3), and the emotion labels and analysis are the authors' own; the encumbrance above attaches to the vector tensors as weight-derived Model Derivatives.

The grid's files under gemma_grid/ are Gemma Model Derivatives of the six Gemma 2 checkpoints (2B, 9B and 27B, base and instruction-tuned), distributed under the same terms. gemma_grid/NOTICE lists them.

License

gemma — Gemma Terms of Use. The vectors are a Gemma Model Derivative (see "Gemma provenance and redistribution" above), so redistribution of the artifacts in this repository is governed by the Gemma Terms of Use; the §3.2 use restrictions travel downstream, and the NOTICE, GEMMA_TERMS.md, and Prohibited Use Policy apply. This is the governing license for the repository.

Anthropic dimension (provenance, not a redistribution blocker). The vectors were computed over Claude-Sonnet-4.5-generated stimuli; under Anthropic's Commercial Terms §B the customer owns those Outputs, and the vectors are a difference-of-means derivative that contains no Claude text. The underlying 205,200-story corpus is not redistributed here — it is released as a generation recipe (see the code repo). A clarification request to Anthropic on the corpus went unanswered; the corpus therefore stays recipe-only (not redistributed), which is the conservative reading of the terms above.

google/gemma-2-27b itself remains subject to the Gemma Terms of Use independently of this repository.

Citation

@misc{hollowell2026replicating,
  title         = {Replicating the Geometry of Emotion Representations in a Base Open-Weights Model},
  author        = {Hollowell, Adam},
  year          = {2026},
  eprint        = {2609.22208},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2609.22208}
}

@misc{hollowell2026grid,
  title         = {The Geometry of Emotion Representations Across Scale, Depth, and Instruction Tuning},
  author        = {Hollowell, Adam},
  year          = {2026},
  eprint        = {2610.35411},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2610.35411}
}