modulation-lens-meta-model-qwen36-27b

Modulation-lens meta-model (Qwen3.6-27B, layer 42)

A LoRA that reads a layer-42 activation and writes the short phrase that state is holding in mind. Trained by GRPO against a geometric reward: the phrase is placed in a modulation template, the model writes a fixed carrier sentence, and the state induced while writing is compared to the activation you fed in.

iter_000375 of the plain-cosine arm (the last checkpoint; the loop saved every 25 steps and the run ended at 400).

The prompt is part of the lens

The adapter is trained for exactly this prompt with exactly this injection. Change either and the readout degrades. Verbatim:

You are shown an internal activation vector captured from a language model at a single position while it was reading some text. The vector is enclosed in <concept> tags.

<concept>㈜</concept>

Your job: write the short phrase that this state is holding in mind.

How it is judged. Your phrase is placed into a prompt of the form
  Focus on the following idea: "<your phrase>" while writing the following phrase: "<a fixed unrelated sentence>"
The model then writes that fixed sentence, and we read its internal state while it does so. You score well when that state matches the state you were given.

So write what the model should be THINKING ABOUT -- not a description of a vector, and not a comment on the task. Natural, fluent English. At most 16 tokens. Output only the phrase.

Rendered with apply_chat_template(..., add_generation_prompt=True, enable_thinking=False).

Injection recipe

• The activation replaces the residual stream at the single marker token ㈜ (id 158983) at decoder block 1.
• The marker must sit inside <concept>...</concept>: the writer verifies the left/right neighbours (ids 29, 510) and raises if no correctly-neighboured marker is found, because a silent no-op is how a run trains for hours on an uninjected prompt.
• Replacement writes a vector ~3.8x the local block-1 norm. Karvonen norm-matched addition (h + ||h||*v/||v||) was trained as a controlled arm and is indistinguishable: block-42 states match at cos 0.982, SFT loss within 1.6%, and probe curves agree to +-0.0007 at matched steps. Either recipe works; use the one you train with.

Sampling

do_sample=True, temperature=1.0, top_p=1.0, top_k=0, max_new_tokens=16, and no min_new_tokens floor -- forcing a minimum blocks the natural EOS on a short phrase and produces run-on junk. 100% of rollouts terminate naturally at ~14.5 tokens; raising the cap to 48 does not change the output, so 16 is not a binding constraint.

What it does

Held-out readouts (blogpost punctuation positions, base model, chat-native):

It converges on slash-separated multi-concept phrases. That is not reward hacking: scored against mismatched targets, discrimination rises from 0.2113 (SFT init) to 0.3198 after RL, i.e. bare grows much faster than mis. Readouts also do not preferentially fire on same-type positions, so they are not merely announcing the text's genre.

workspace-bench (11 families, strict Opus judges)

Read this as specialisation, not a ranking. The oracle lens is the better generalist; this lens is the best of every lens tested at reading directed thought -- which is its own construction ("think about X while writing Y") -- and it wins typo. Relational binding is unsolved by every lens tested, this one included, at or below chance.

Scoring geometry (if you want to reproduce the reward)

Two-mean centring is load-bearing. Candidates are centred by the grid's own filler mean (PMU), targets by the activation-pool mean (AMU); both are fixed constants, never a batch mean. One shared mean makes a blank string score 0.259; two means take it to 0.008. Score in J-space (x @ J[42].T); plain cosine, not whitened -- whitening rewards verbatim copying over characterisation.