template-lens-qwen3-4b

Template lens — Qwen3-4B

An implementation of template lens for Qwen/Qwen3-4B, from [Verbalizable Representations Form a Global Workspace in Language Models](https://transformer-circuits.pub/2026/workspace/index.html. Using a predefined vocabulary, replicates J-Lens to an extent on multi-token concepts.

<table> <tr> <td width="33%"><a href="https://cdn-uploads.huggingface.co/production/uploads/6997e2c73023436326a52378/SLEbdTRjth8zgSUCMZTmi.png"><img src="https://cdn-uploads.huggingface.co/production/uploads/6997e2c73023436326a52378/SLEbdTRjth8zgSUCMZTmi.png" alt="Example 1"></a></td> <td width="33%"><a href="https://cdn-uploads.huggingface.co/production/uploads/6997e2c73023436326a52378/0H1xmDksClV_F0zMvIrXJ.png"><img src="https://cdn-uploads.huggingface.co/production/uploads/6997e2c73023436326a52378/0H1xmDksClV_F0zMvIrXJ.png" alt="Example 2"></a></td> <td width="33%"><a href="https://cdn-uploads.huggingface.co/production/uploads/6997e2c73023436326a52378/kOy0r1w0DuT1gC46wH3Rf.png"><img src="https://cdn-uploads.huggingface.co/production/uploads/6997e2c73023436326a52378/kOy0r1w0DuT1gC46wH3Rf.png" alt="Example 3"></a></td> </tr> </table>

This repo includes the complete fitted lens (36 layer tensors), the matching 16,213-word vocabulary, fitted moments, and the generated and mined datasets retained from development.

Released lens

The per-layer files total about 2.99 GB; a single layer is about 83 MB. Download only the layers needed. The 3.93 GB moments file is needed for inspecting the fit or changing the ridge, not for ordinary readout.

Method

For each word w, collect contexts ending immediately before that word would naturally occur, and average the final-position activation at block l:

t_w(l) = (Sigma_l + lambda_l I)^-1 (mu_w(l) - mu(l))
lambda_l = 3 * trace(Sigma_l) / 2560

The mean and covariance use per-passage weights 1 / P_w, where P_w is the number of passages for word w, so vocabulary entries contribute equally despite unequal passage counts. The covariance solve uses float64 Cholesky; stored readout matrices are float16. The saved vectors are*not unit-normalized.

For a raw residual direction v, score the matching layer's templates. Raw projection is t_w · v. The provided helper defaults to cosine scoring (unit-normalizing both vectors) to reduce domination by large-norm templates. Use normalize=False to recover the raw projection rule. Rank -v separately when investigating the negative direction. Scores are similarities, not calibrated probabilities. Do not whiten v again: the covariance correction is already in the template vectors.

Minimal use

Install PyTorch and huggingface_hub. No base-model weights are needed if you already have a residual vector.

import importlib.util
import torch
from huggingface_hub import hf_hub_download

repo = "andyx10/template-lens-qwen3-4b"
helper = hf_hub_download(repo, "code/read_template.py")
spec = importlib.util.spec_from_file_location("read_template", helper)
lens = importlib.util.module_from_spec(spec)
spec.loader.exec_module(lens)
words, templates = lens.load_layer(18)

v = torch.load("activation.pt", map_location="cpu", weights_only=True)  # [2560]
print(lens.rank_direction(v, words, templates, top_k=15))

Each .pt file is a tensor dictionary with keys layer, ridge_c, lambda, final_norm and templates; it loads with torch.load(..., weights_only=True). Rows follow the exact order of vocab.json["words"]. For block 35, capture the residual before final RMSNorm; the model's last output_hidden_states entry can already be normalized. Use a block-output hook or the supplied training capture helper. Pin a HF commit through revision for reproducible downloads.

Vocabulary and data provenance

The released content-v1 vocabulary keeps 8,213 entries from the earlier 12,744-word vocabulary and adds 8,000 nouns, using frequency rankings and WordNet part-of-speech filtering. The extension reaches frequency rank 34,418.

This vocabulary is not fully topic-blind, and the he original vocabulary included 44 words added from evaluation-cohort topics. The content filter protected 188 topic words, 38 of which would otherwise have been pruned. These choices are recorded in vocab.json and data/vocabulary/content-filter.json. Do not describe downstream evaluation as independent of vocabulary construction. Full topic strings in ceiling_vocab.json remain separate diagnostic candidates; they are not the released template-row vocabulary.

The main passage collections were generated with DeepSeek V4 Pro through OpenRouter. The release keeps both the base collection and content-word extension, along with a corpus-mined pilot, held-out pilot contexts and vocabulary bake-off artifacts. The final fit uses the relevant passages for the final vocabulary; the base passage file alone is not the complete fitting corpus.

Passage JSONL rows use w (lowercase vocabulary word), s (surface form) and t (context ending before the target). Generation was designed to exclude the target word from its context. These texts are synthetic fitting inputs, not human-verified factual statements. See data/README.md.

Repository layout

lens_config.json / vocab.json                   released configuration and row vocabulary
templates/                                     36 fitted layers + original fit metadata
statistics/moments.pt                           means, covariance and per-word counts
data/
  passages/base/                               original generated corpus + vocabulary metadata
  passages/content-extension/                  additional generated contexts + target words
  vocabulary/                                  content filter, earlier vocabularies, bake-off
  pilot-mined/                                 mined train/held-out contexts and pilot diagnostics
training/logs/                                 original generation logs
code/read_template.py                          lightweight loading and ranking helper
code/training/                                 standalone fitting/capture pipeline source
provenance/                                    full original-to-current path/hash map
archive/README-before-cleanup.md                superseded model card, retained for provenance

The moments dictionary contains word_mu [16213,36,2560], word_n [16213], mu [36,2560], sigma [36,2560,2560], the word list and fit metadata. The field sigma stores the covariance; it is not an already inverted whitening matrix. The original templates/meta.json uses [0,35] as an inclusive block range; lens_config.json lists every block explicitly.

All original datasets, fit files and diagnostics are retained byte-for-byte. provenance/file-map.json records every path change. The old layout is available at commit 258460d66936033785655597eb976a07f8ce01cc (tag legacy-layout-v1) for scripts that expect content-v1/templates-c3/ or root passages.jsonl.

Scope and limitations

This is an independent reconstruction of the method, not an official authors' checkpoint. It reads associations over a fixed, English vocabulary; a concept outside that vocabulary cannot be named directly. Passage generation, frequency selection, topic-word additions and unequal per-word sample sizes all affect the readout. Normalization changes ranking and must be reported in evaluations.

The release records the available fitted artifact at ridge c=3; this is not a claim of universally optimal ridge or a new benchmark result. Pilot next-word diagnostics refer to earlier configurations and are labeled under data/pilot-mined/. Base-model and dataset revision pins were not fully retained for the original fit. Dataset and third-party source terms remain applicable separately from this repo's Apache 2.0 license.

The companion oracle lens generates free-form concepts and uses its own on-policy whitening statistics. These template moments are not a replacement for the oracle's whitening files.

License

Released under the Apache License 2.0.