jacobian-lens-deepseek-v4-flash-preview

Jacobian lens for DeepSeek-V4-Flash (preview)

Fitted Jacobian-lens matrices for deepseek-ai/DeepSeek-V4-Flash — the April 2026 preview release that DeepSeek-V4-Flash-0731 officially superseded ("same model architecture and size … only re-post-trained"). The Jacobian lens (Anthropic, Verbalizable Representations Form a Global Workspace in Language Models) reads out what an internal residual is disposed to make the model say:

lens_l(h) = unembed( J_l @ h ),   J_l = E[ ∂h_target / ∂h_l ]

There is no training and there are no learned parameters: each J_l is a closed-form running mean of per-prompt Jacobians.

Why this lens exists

Together with the 0731 lens, this is one arm of a recipe-controlled comparison: one reported base (vendor-described — DeepSeek states the 0731 release "was only re-post-trained"), two post-trainings, matched fitting recipes. The headline finding is that the second post-training changed the token-level lens readout far more than the lens geometry — cross-checkpoint top-1 readout overlap is 0.293 against a same-checkpoint disjoint-corpus floor of 0.592, while mean CKA is 0.821 against a floor of 0.921 — with the band, ignition dynamics, selectivity, white-bear suppression and J-space compression all unchanged. Details and controls: see the write-up linked below.

What the matrices are

V4-Flash carries hc_mult = 4 parallel residual streams per position (hyper-connections), so a square J_l ∈ R^{d×d} is not well-defined. These lenses are the rectangular generalization: source is the four flattened streams (16384), target is the hc_head stream-collapse output (4096), so the collapse operator is absorbed into J_l.

Files

The band, corpus (same WikiText-103 stream, first 100 records ≥600 chars, 128 tokens), skip_first=16, per-prompt normalization, and dim_batch are all identical to the 0731 n=100 lens, so any preview-vs-0731 lens difference is attributable to the post-training change rather than to fitting settings. The chat encodings of the two checkpoints were verified byte-identical for the modes used before any comparison was made.

Usage

Requires the mHC-capable (rectangular-J) version of the jlens package:

import transformers, jlens

name = "deepseek-ai/DeepSeek-V4-Flash"
hf = transformers.AutoModelForCausalLM.from_pretrained(name, dtype="auto", device_map="auto")
tok = transformers.AutoTokenizer.from_pretrained(name)
model = jlens.from_hf(hf, tok)

lens = jlens.JacobianLens.from_pretrained("xiangchensong/jacobian-lens-deepseek-v4-flash-preview")

Citation

• Method: Anthropic, Verbalizable Representations Form a Global Workspace in Language Models, Transformer Circuits, 2026. https://transformer-circuits.pub/2026/workspace/index.html
• This lens / the recipe-controlled comparison: The Jacobian Lens at Frontier Scale: Testing the Global Workspace on DeepSeek-V4-Flash and GLM-5.2. https://xiangchensong.github.io/blog/2026/jacobian-lens-global-workspace/

Code: https://github.com/xiangchensong/jacobian-lens-open-frontier