Downloads · 30 days
19
12% of all-time downloads
praxagent-org/jacobian-lens-qwen3.5-397b-a17b
jacobian-lens-qwen3.5-397b-a17b is a machine learning model from praxagent-org. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
A fitted Jacobian lens (Anthropic's jlens, Apache-2.0) for Qwen/Qwen3.5-397B-A17B — to our knowledge the first publicly available Jacobian lens for a ~0.4T-parameter model (the Neuronpedia collection spans 70M–70B). F…
Downloads · 30 days
19
12% of all-time downloads
All-time downloads
158
Public
Repo size
2 GB
Likes
0
Public
Click a slice to open those files.
.pt2 GB · 100%
From the Hugging Face model README
A fitted Jacobian lens (Anthropic's jlens, Apache-2.0) for Qwen/Qwen3.5-397B-A17B — to our knowledge the first publicly available Jacobian lens for a ~0.4T-parameter model (the Neuronpedia collection spans 70M–70B). Fitted and validated by praxagent as part of an independent replication/audit of Anthropic's Verbalizable Representations Form a Global Workspace in Language Models (2026).
The layer×layer CKA of the lens's token geometry shows three depth regimes, with a mid band whose separation is the largest we have measured across 36 public lenses:
mid_sep = +0.3434 (n=24; interim n=16 read +0.3796)
within early/MID/late CKA = 0.890 / 0.937 / 0.814 (n=24)
Two controls travel with that number (see the atlas note for both): a Frobenius-matched random-transport null is featureless (mid_sep -0.000113), and a distance-only surrogate with the same similarity-versus-layer-distance profile reproduces 0.275 of the 0.343, so about a fifth of the statistic is block structure beyond smooth decay with layer distance. On that fitted statistic the 397B's excess over its surrogate (+0.125) is the largest among the 36 lenses. Cross-model comparisons of the band statistic are only meaningful on a shared probe (own-vocabulary values, including an earlier "1.6x Qwen3-14B" comparison in this card, are not comparable across models and have been withdrawn); on the shared probe the 397B's mid_sep is 0.386 and its fitted separation 0.472, the highest in the collection. We make no claim that the band grows with scale: across the collection banding tracks size only loosely (rank correlation about +0.4 to +0.5).
| file | what |
|---|---|
jlens/wikitext/qwen35_397b.pt | fp32 lens (canonical jlens format, JacobianLens.load-able) |
jlens/wikitext/qwen35_397b_fp16.pt | fp16 Jacobians ({"J": {layer: tensor}, "source_layers": [...]}), half the size |
band.json | the band statistic + per-block CKA numbers |
evals.json | held-out lens-fidelity evals (below) |
ignition_qwen3.5-397b.json | workspace-extraction (ignition) results run through this lens |
SHA256SUMS | checksums, matching the fit machine's originals |
fit() — J_l = E[∂h_final/∂h_l] for each of the 59 source layers, indices 0–58 (every decoder layer except the last). Qwen3.5-397B-A17B has 60 layers total (0–59); layer 59 is the target h_final that the lens transports toward, so it is not itself a source. The artifact holds 59 Jacobians (source_layers = [0, 1, …, 58]). The workspace-readout demos use the middle-third band: layers 19–38 (20 layers).Salesforce/wikitext, seed 0), max_seq_len 128,
dim_batch 16Qwen3_5MoeForConditionalGeneration (⚠️ NOT AutoModelForCausalLM,
which silently mismatches this checkpoint's model.language_model.* keys), text
backbone via jlens.Layout(path="model.language_model"), bf16,
attn_implementation="eager", device_map="auto" with an even 110 GiB/GPU cap,
on 8×H200. ~10 min/prompt.corpus_dependence).evals_v2_397b.json, calg2_neuronpedia_calibration.log. (An earlier eval,
evals_v1_misspecified.json, used absolute agreement — the wrong metric — and is
retained only as a transparency receipt.)ignition_qwen3.5-397b.json.668c3bf1…99e97, byte-identical to the fit
pod's original) plus only the lm_head shard of the base model, and recomputed the
band statistic from the public copy: mid_sep = +0.343363, agreeing with the
shipped band.json to 2×10⁻⁸. Runnable protocol:
experiments/fit_our_own/consumer_check_397b.py in the repo below.experiments/lens_demo/ in the repo below.# pip install torch safetensors numpy huggingface_hub
# 1) download the fp16 lens from this repo + lm_head shard from Qwen/Qwen3.5-397B-A17B
# 2) J_l = lens["J"][l].float(); probe 4096 vocab rows (numpy default_rng(0))
# 3) geometry_l = U_probe @ J_l ; layer-x-layer linear CKA; band statistic
# full script: experiments/fit_our_own/consumer_check_397b.py in the repo below
model.language_model); the vision tower
and MTP heads are untouched.License: Apache-2.0 (matching both the base model and jlens). Please cite/attribute
praxagent and link the repo if you build on this artifact.
The full 59x59 centered-kernel-alignment matrix behind the released band statistic,
recomputed via the hash-verified consumer path (this repo's lens + the base model's
lm_head, seed-0 n_probe=4096): cka/cka_397b.npz + heatmaps. Correctness gate: the
matrix reproduces the released mid_sep +0.343363 to |delta| = 2e-8. A Frobenius-matched
random-transport control (cka/cka_397b_null.png) is structure-free (mid_sep -0.000):
the early / mid-band / late block structure comes from the fitted lens, not from the
shared unembedding. That null is necessary and not sufficient: a distance-only surrogate
(every cell replaced by the mean CKA at that layer distance) reproduces 0.275 of the 0.343, so
read the block structure as the fifth of the statistic that exceeds smooth decay with distance
(fitted separation 0.407 real against 0.282 surrogate). Correction history: this card's earlier
"1.6x Qwen3-14B", "grows to frontier scale", "safely converged" and "near-perfectly" wordings
were withdrawn on 2026-09-06 after our own later measurements (36-lens atlas, fit-budget sweep,
8B extension); the previous text is in this repository's commit history. Generator:
projects/jacobian-lens-and-identifiability/experiments/fit_our_own/cka_heatmap_397b.py
in the research repo (commit c6c7bb1).