template-lens-qwen3.6-27b

Template lens — Qwen3.6-27B

Two fitted template lenses for one model: a 12-word pilot (the method proven, Fig-62 replicated) and a 175-word deception probe panel with a calibrated reader. Contents are described per release below.

Release 1 — pilot (12 words)

A fitted template lens for Qwen/Qwen3.6-27B, implementing the method from the appendix "Extending the Jacobian lens to multi-token concepts" of Anthropic's Verbalizable Workspace paper. Correction, 2026-09-23. The first version of this README said this was the first public template-lens artifact for any model. That was wrong. camilablank/workspace-lenses has carried a template lens for this same model since 2026-08-10: 13,174 rows, all 64 layers, cosine scoring, with build code at camilablank/template-lens. That is nine days before this pilot. My survey on 2026-08-19 missed it, and I repeated the claim for a month. The claim is withdrawn. This repo is a small, gated fit of the same appendix method. What it adds is the held-out readout gate, the Fig-62 replication with numbers, and the 175-word deception and honesty probe panel with controls. It was replicated on 2026-09-23 on Qwen3.6-35B-A3B and Qwen-AgentWorld-35B-A3B; those artifacts will be published separately.

For a word w, the template vector is the ridge-regularized linear discriminant for "the model is about to say w":

t_w(l) = (Sigma_l + lambda I)^-1 (mu_w(l) - mu(l))

where mu_w(l) is the mean residual-stream activation at the final position of passages written so that w is their natural continuation (and never contains w), and mu, Sigma are the mean/covariance over all training passages at layer l. Read a concept by projecting a residual onto t_w; steer by adding/subtracting; no training loop, no gradients.

Contents

Vocabulary (leading-space token counts verified on this model's tokenizer): fire, Paris, mirror, blackmail, consciousness (1 tok); photosynthesis, Beethoven, New Zealand (2); self-awareness, Golden Gate Bridge (3); Tchaikovsky, eavesdropping (4).

Pilot results (all numbers reproducible from this repo)

Held-out readout (gate 1). Top-1 word identification among 12 (chance 0.083) climbs monotonically with depth: 0.31 (embeddings) → 0.63 (L12) → 0.83 (L36) → 0.979 (final layer).

Functional comparison vs an Anthropic-fit J-lens (gate 3). Using the Neuronpedia-hosted J-lens for this model (n=1000, credited to Anthropic Interpretability), top-hit rate by concept token length on the same held-out residuals, layer 60:

The J-lens is healthy on single tokens and degrades sharply with concept length; the template lens stays flat — the paper's core claim, replicated. Note the ranking universes differ (12-way vs full vocab); the shape contrast is the result, not the absolute gap.

Geometric alignment (gate 2). Median cosine between single-token templates and J-lens directions J_l^T u_t is 0.115 (per-word maxes 0.18–0.33). Strongly non-random in 5120 dims (~8–24σ) but far from vector identity — consistent with the paper's footnoted deviations (different averaging corpora; logit- vs probability-gradients; ridge).

Swap steering (gate 4) — partial, reported honestly. Using a simplified intervention (h += gamma*(mu_B − mu_A) at a layer band; NOT the paper's projective coefficient-matched swap): Tchaikovsky→Beethoven reproducibly flips "The composer of Swan Lake… was born in the country of" from Russia to Germany (two band/scale combos) or Austria/"Ludwig". The reverse direction injects Tchaikovsky content ("…the composer of the Nutcr…") without flipping the answer, and a fire→mirror null control contaminates generation — so this translation is blunt concept injection, a causal handle but not a clean swap. The projective swap is the natural next step.

Release 2 — deception probe panel (175 words, 2026-08-20)

A bucketed probe panel: 62 deception + 12 AI-deception + 25 honesty (the opposite pole) + 48 neutral controls + the 28 pilot words as a regression set. Controls and the honesty pole are what make readings interpretable. Full report: RESULTS_panel_20260820.md.

Held-out (1,400 passages): word top-1 0.31 / top-10 0.52 at L60 (175-way, chance 0.006); bucket top-1 0.65 (chance ≈ 0.25). A coarse instrument by design — synonyms blur; families separate.

Whole-prompt readout (the paper's multi-hop result, on this panel): reading every prompt position, all six calibration latents rank 1/175 at their cue tokens, and four abstract latents never named in the prompt read at z 10–16 (procrastination, deja vu, eavesdropping, vertigo 16.0); controls show nothing in the pilot bucket. Reading only the four pre-answer template positions was a null — read where the concept is inferred, not where the answer is typed, and mask position 0 / special tokens.

Known confound: passage neighbors. Templates built from authored lead-up passages carry the words that tended to sit near the target in those passages, so a readout for silence also lifts its passage neighbors, and target-versus-neighbor comparisons are unreliable. On this panel, z-scoring each word against its own null over ordinary text (out_panel/null_calibration.safetensors) reduced the neighbor words substantially. Small panel; treat as a hint for anyone building the full vocabulary.

Audit-driven caveats: 13 templates are weak (the subject rarely ranks the target in its top-50 at their passage endpoints: infiltrate, scheming, duplicity, …) and fire idiosyncratically; readers filter them. Filtering the training passages to subject-endorsed ones and refitting made accuracy worse (too few passages left) — kept as a negative result. Polysemy leaks (bluff fires at "cliff"). Nothing here yet separates talking about deception from being deceptive; that requires matched content with varied intent.

Reader

python panel_chat.py --model <Qwen3.6-27B path> --out_dir out_panel --layer 60 --z 3.5

Each reply prints the panel words that fired as word(max-z × tokens≥z) by bucket. panel_battery_trace.py --prompts file.txt reads every position of prompt+reply at chosen layers and writes a transcript-style report.

Usage (release 1 artifacts)

import json, torch
from safetensors import safe_open
from safetensors.torch import load_file

d = load_file("out/templates.safetensors")
with safe_open("out/templates.safetensors", "pt") as f:
    meta = f.metadata()
layers = json.loads(meta["layers"]); words = json.loads(meta["words"])

# score a residual-stream activation h (torch [5120], hidden_states index l)
li = layers.index(44)
scores = d["templates"][li] @ h          # [12] — one score per word
print(sorted(zip(words, scores.tolist()), key=lambda x: -x[1])[:3])

To mint a template for a new word: generate passages the same way (passages/ shows the format), collect final-position residuals, and solve against out/sigma.safetensors — no model fitting required beyond forward passes.

Provenance

Release 1: fit 2026-08-19 on 1× RTX PRO 6000 Blackwell 96 GB (vast.ai), bf16, 624 training passages, 134 s; passages written by Claude (Fable 5). Release 2: fit 2026-08-20, same hardware class, 9,800 training passages, ~13 min; passages written by OpenAI gpt-5.4 (reasoning none) and audited against the subject's own next-token ranks. Details in the fit_provenance*.json files.

Method: Anthropic, Verbalizable Representations Form a Global Workspace in Language Models (2026), appendix. Cross-check lens: neuronpedia/jacobian-lens (@mntss). This artifact: ec75hash, with Claude Code.