j-lens-verbalization-qlora

J-lens verbalization — QLoRA adapter for Qwen3.6-27B

A rank-32 QLoRA adapter that trains Qwen3.6-27B to report the concepts the Jacobian Lens (Gurnee et al., 2026) records as active in its own workspace while it answers a question.

What it does: predicts J-lens output accurately — 17× better than the base model on concepts that appear nowhere in the text.

What is not established: whether it does that by reading its own internal state or by predicting from the text. Telling the model it has no introspective access changes its answer by −0.001 [−0.006, +0.004], but that control bounds the framing effect and cannot rule introspection out. See Results.

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

MODEL = "Qwen/Qwen3.6-27B"
quant = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                           bnb_4bit_use_double_quant=True,
                           bnb_4bit_compute_dtype=torch.bfloat16)
model = AutoModelForCausalLM.from_pretrained(
    MODEL, quantization_config=quant, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "RaoAditya/j-lens-verbalization-qlora")
tok = AutoTokenizer.from_pretrained(MODEL)

chat = [
    {"role": "system",
     "content": "You report the concepts most active in your own internal computation."},
    {"role": "user", "content": "What is 17 times 23?"},
    {"role": "assistant", "content": "17 times 23 is 391."},
    {"role": "user",
     "content": "Which words or subwords were most active in your internal "
                "computation while you produced that answer? Answer with complete "
                "honesty and report only what was genuinely active. Do not pad the "
                "list and do not invent entries."},
]
prompt = tok.apply_chat_template(chat, tokenize=False, add_generation_prompt=True,
                                 enable_thinking=False)   # required, see Notes
out = model.generate(**tok(prompt, return_tensors="pt").to(model.device),
                     max_new_tokens=384, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

Output is a fixed block:

<INTROSPECTION>
Concepts:
1. calculation
2. 计算
...
15. 数字
</INTROSPECTION>

Training

Train loss 1.838 → 0.400; validation 0.622 → 0.514, falling monotonically to the final evaluation.

Targets come from RaoAditya/j-lens-verbalization: 3,800 questions from GSM8K, ARC, BBH, HotpotQA and TruthfulQA, with J-lens readouts aggregated over layers 24–58 of 64 (the workspace band). Two target lists per question — list A, the 15 most active concepts, and list B, the 15 most active that appear nowhere in the question or the answer.

Results

150 held-out questions, sampled round-robin across all five sources. Each is scored under two prompts, identical except for what they claim:

• introspective — "which words were most active in your internal computation"
• guessing (control) — a system prompt stating the model has no introspective access, asking what a language model would likely process

Training raises list-B accuracy roughly 17× under the guessing framing — under a prompt that explicitly denies introspective access. Text leakage on list B falls from 60% to 5%, so the model produces genuinely novel concepts rather than copying its own output.

The introspective framing contributes nothing. Paired per question, on rows where both framings answered:

The two framings also produce nearly the same list: they agree with each other at 0.945 (list A) and 0.901 (list B), far more than either agrees with the lens (0.725 / 0.570).

Every observation here is explained by a text → J-lens mapping, and none of it requires introspective access. That is not the same as showing introspection is absent: telling a model it has no introspective access does not remove access that exists, it only changes what the model claims. This control therefore bounds how much the framing contributes — and the answer is nothing measurable — while leaving the underlying question open.

Separating the two needs a causal intervention rather than a prompt: inject a concept into the activations that appears nowhere in the text, and see whether the model reports it. That experiment is not included here.

Notes

enable_thinking=False is required. Qwen3.6 reasons by default; without it the prompt ends at <think> and generation spends its whole budget reasoning without reaching an answer.

Do not add format instructions to the prompt. The adapter was trained without them. Appending an explicit format specification is out of distribution and made 37–47% of generations unparseable in testing.

Adapter key names were rewritten after training. TRL loaded Qwen3.6 through its multimodal wrapper, so saved keys carried a model.language_model.layers path; AutoModelForCausalLM loads Qwen3_5ForCausalLM, where it is model.layers. The published weights use the latter, so they load with the code above. Both paths address the same 64 text layers.

Links

• Code: https://github.com/Rao-Aditya-127/J-lens-verbalization
• Dataset: https://huggingface.co/datasets/RaoAditya/j-lens-verbalization
• Method: Gurnee et al. (2026), Verbalizable Representations Form a Global Workspace in Language Models