jacobian-lens-glm-5.2

Jacobian lens for GLM-5.2-FP8

Fitted Jacobian-lens matrices for zai-org/GLM-5.2-FP8 (753B total / ~40B active MoE, 78 layers, hidden size 6144, MLA + DSA sparse attention). The Jacobian lens (Anthropic, Verbalizable Representations Form a Global Workspace in Language Models) reads out what an internal residual is disposed to make the model say:

lens_l(h) = unembed( J_l @ h ),   J_l = E[ ∂h_final / ∂h_l ]

There is no training and there are no learned parameters: each J_l is a closed-form running mean of per-prompt Jacobians.

What the matrices are

GLM-5.2 has a conventional single-stream residual, so the standard square lens applies:

• Shape: [6144, 6144] per layer, fp16 on disk.
• Source: the output of decoder block l; target: the output of the final decoder block (layer 77), before the final norm and unembedding.
• Layers: L34–71 (38 layers), the model's global-workspace band. The band was projected from the relative depth of the band measured on other families and confirmed by onset/offset statistics on this model (top-1 readout accuracy 0.0012 at L34 rising to 0.194 at L71).

What this lens measures — fp8, straight-through. GLM-5.2 is served in native fp8, and dequantizing to bf16 (~1.5 TB) exceeded available VRAM, so the fit ran against the fp8 kernels directly. Those kernels register no autograd formula; backward formulas were added for the three w8a8_block_dynamic matmul ops — activation gradients only, weights frozen, straight-through with respect to the kernels' internal activation quantization. The Jacobian here is therefore of the fp8 model as served, which is arguably the object of interest, but it is not the Jacobian of a bf16 dequantization.

File

Fitting recipe

• Corpus: WikiText-103 (Salesforce/wikitext, wikitext-103-raw-v1, train split, streamed), records ≥ 600 characters, truncated to 128 tokens.
• Positions: the first 16 positions are skipped (skip_first=16, attention-sink positions) and the final position is dropped; the per-prompt Jacobian is the mean over the remaining ~111 source positions, with cotangents set at every valid target position at or after the source.
• Per-prompt normalization (normalize_per_prompt=True): each prompt's J_l is scaled to unit Frobenius norm before averaging (per-prompt gain is heavy-tailed and the final RMSNorm makes uniform rescaling of J_l invisible in the logits). Consequence: ||J_l||_F ≤ 1 by construction (measured 0.31–0.95 across the band) — only the direction of J_l is meaningful, not its absolute scale.
• n=100: on DeepSeek-V4 the estimator was measured to be converged from roughly n=60, and an n=1000 refit moved nothing beyond the corpus-sampling noise floor; the same corpus size was used here.
• Storage: fp16; loaded back as fp32.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from jlens.hf import from_hf
from jlens.lens import JacobianLens

lens = JacobianLens.from_pretrained("xiangchensong/jacobian-lens-glm-5.2")

name = "zai-org/GLM-5.2-FP8"
tok = AutoTokenizer.from_pretrained(name)
hf_model = AutoModelForCausalLM.from_pretrained(name, torch_dtype="auto", device_map="auto")
model = from_hf(hf_model, tok)

lens_logits, model_logits, input_ids = lens.apply(
    model, "The capital of France is", layers=[50], positions=[-1]
)
print(lens_logits[50].topk(5).indices)   # what L50 is disposed to say

Readout is forward-only: applying the lens needs no backward pass, so the fp8 autograd patches used for fitting are not required to use it. The lens is square, so it also loads with the upstream (square-only) jlens release.

License and provenance

The matrices in this repository are derived from the base model: each J_l is an expectation of gradients of zai-org/GLM-5.2-FP8's forward pass, i.e. a function of its weights. The base model is released under the MIT License (Copyright (c) 2026 Zhipu AI), and these matrices are released under the same terms. No base-model weights are contained in or recoverable from these matrices at fidelity; they are averaged input–output sensitivities.

Citation

• Method: Anthropic, Verbalizable Representations Form a Global Workspace in Language Models, Transformer Circuits, 2026. https://transformer-circuits.pub/2026/workspace/index.html
• This replication / the rectangular-J generalization: The Jacobian Lens at Frontier Scale: Testing the Global Workspace on DeepSeek-V4-Flash and GLM-5.2. https://xiangchensong.github.io/blog/2026/jacobian-lens-global-workspace/

Code: https://github.com/xiangchensong/jacobian-lens-open-frontier