jacobian-lens-qwen3-8b

Jacobian Lens for Qwen3-8B

A fitted Jacobian lens (J-lens) for Qwen/Qwen3-8B, from Anthropic's Verbalizable Representations Form a Global Workspace in Language Models, fitted with the reference implementation anthropics/jacobian-lens.

Contents

qwen3_8b_lens.pt — a torch.save dict with keys J (dict layer -> [4096, 4096] fp16 tensor), n_prompts, source_layers, d_model. Load it with jlens.JacobianLens.load(...).

Usage

import torch, transformers, jlens
from huggingface_hub import hf_hub_download

hf = transformers.AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-8B", dtype=torch.bfloat16
).to("mps")  # or "cuda"
tok = transformers.AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
model = jlens.from_hf(hf, tok)

path = hf_hub_download("dormantx/jacobian-lens-qwen3-8b", "qwen3_8b_lens.pt")
lens = jlens.JacobianLens.load(path)

# per-layer lens logits at the last position, J-lens vs logit-lens baseline
lens_logits, model_logits, _ = lens.apply(
    model, "Fact: The currency used in the country shaped like a boot is",
    positions=[-1], layers=[16, 20, 24], use_jacobian=True,
)

License

Apache-2.0, matching the reference implementation. The base model Qwen/Qwen3-8B is subject to its own license.