qwen3.5-4b-mean-future-r-lens

Qwen3.5-4B Mean-Future R-Lens

A fitted readout tool for transformer internals. It maps an earlier-layer activation into Qwen3.5-4B's layer-30 residual basis for vocabulary inspection. It is not a language model or adapter and does not generate text by itself.

Overview

The release contains 30 float32 matrices—one for each source layer 0–29—each with shape 2560 × 2560. For a row-vector activation x_l:

x_layer_30_basis = x_l @ R_l.T

Apply Qwen's existing final normalization and unembedding to inspect the vocabulary ranking. The matrices average RelP pseudo-Jacobians produced by a modified backward graph; they are not ordinary Jacobians of Qwen's forward computation. Model weights, activations, and the forward pass remain unchanged.

Token rankings are interpretive projections, not proof that a layer stores literal words or that an early decoded token is the model's final answer. Rare or weakly trained vocabulary rows can also mislead. See the original R-Lens post for motivation and broader caveats.

Mean-Future estimator

The fit uses every source position except the final token. For source position s, valid targets are the non-final positions t ≥ s.

• Sum R-Lens: adds all valid future-target contributions, giving earlier positions more total target weight.
• Mean-Future R-Lens: divides by the number of valid targets before averaging across positions and prompts.

This removes the deterministic future-horizon count factor without forcing equal map norms.

<details> <summary>Definition and adjoint</summary>

For sequence length $L$, let $S={0,\ldots,L-2}$ and $T_s={t\in S:t\ge s}$. With RelP pseudo-Jacobian block $J^{\mathrm{RelP}}_{l,s\rightarrow30,t}$:

$$ R^{\mathrm{sum}}{l,s}=\sum{t\in T_s}J^{\mathrm{RelP}}{l,s\rightarrow30,t},\qquad R^{\mathrm{mean}}{l,s}=\frac{1}{|T_s|}\sum_{t\in T_s}J^{\mathrm{RelP}}_{l,s\rightarrow30,t}. $$

The normalized position maps are averaged within prompts and then across prompts. The adjoint pullback is:

source_scoring_direction = R_l.T @ target_scoring_direction

It pulls a scoring covector—such as an unembedding row or token contrast—toward a source layer. It is not an inverse reconstruction of a source residual.

</details>

Run the included aggregation example with python examples/aggregation_demo.py.

Evidence

Held-out Mean-Future vs Sum

The normalization choice was tested under a frozen matched protocol on Qwen3.5-4B and 9B: Pile fit prompts, separate WikiText-103 evaluation records, four independently fitted conditions per model, and separate primary/replication splits. All pooled 95% intervals for the preregistered summaries favored Mean-Future:

Both model sizes passed all transport and causal condition gates. Lines below are pooled estimates; bands are 95% condition-stratified prompt-bootstrap intervals; shading marks the preregistered early-layer band. Layerwise curves are diagnostic, not separate multiple-comparison-adjusted claims.

Exact results and frozen source hashes are in evidence/mean-vs-sum.json and evidence/per-layer-transport.json.

Evidence boundary: the campaign used 6 fit prompts per condition and 4 conditions per model. The uploaded artifact was fitted separately on 100 prompts using all non-final source positions. The campaign validates the Mean-Future normalization, not held-out performance of these exact matrices or a fresh R-Lens-versus-J-Lens superiority claim.

Matched diagnostic comparisons

The following fixed-prompt heatmap compares freshly matched 100-prompt J-Lens, Sum R-Lens, and Mean-Future R-Lens fits using the same model revision, prompts, positions, layers, and target. Cell text is the decoded top-1 token; green means the target top-1 appears in the lens top-10; the adjacent panels report top-10 overlap. This is illustrative, not held-out evidence. Open full-resolution SVG.

Sum R-Lens baseline note: prompts 1–2 used Qwen's native gated-delta implementation; prompts 3–100 resumed from the verified checkpoint with fla-core==0.5.2 and Triton 3.7.1. Backend equivalence was not independently tested. This affects the illustrative Sum baseline, not the released Mean-Future artifact.

The CKA comparison uses linear CKA over 1,200 uniformly sampled rows of W_U @ J_l, an identity anchor at layer 30, and a second independent 1,200-token sample for stability. Open full-resolution SVG.

The full matrix is stable across vocabulary samples (r=0.9998, mean absolute difference 0.0024). Early J/R divergence and late convergence replicate; the Qwen-3.6-27B claim of roughly 4–5 J-space bands versus 2–3 R-space bands does not robustly replicate on Qwen3.5-4B. Exploratory segmentation selects 2 bands for J-Lens and 3 for each R-Lens variant, but band count is method-dependent and the matrices are primary. CKA is fitted-matrix geometry, not causal or held-out behavioral evidence.

Exact heatmap cells and fit identities are in evidence/layer-position-readout.json. Primary/replication CKA matrices, sample hashes, summaries, and segmentation are in evidence/cross-lens-cka.json. The method follows the J-Lens CKA analysis.

Artifact

Milestones were saved after 10, 25, 50, and 100 prompts; the 50-prompt lens has cosine 0.9979 to the canonical 100-prompt artifact. The exact release file also passed a live test against the pinned model: fingerprint matched, all 30 maps produced finite non-identity outputs, and a final-layer positive control reproduced logits with zero maximum absolute error. This synthetic smoke test validates wiring and compatibility, not superiority. See validation.json and evidence/functional-smoke-test.json.

Load and use

pip install -r requirements.txt
python load_lens.py
python load_lens.py --local ./lens.pt --demo --layer 16

lens.pt is a postconcept.lens version 2 PyTorch payload. load_lens.py downloads or opens it, then checks its SHA-256, format, finite 2560 × 2560 matrices, exact model identity, layers, prompt count, and Mean-Future settings using torch.load(..., weights_only=True).

import torch
from load_lens import MeanFutureRLens

lens = MeanFutureRLens.from_hub("esceptico/qwen3.5-4b-mean-future-r-lens")
x = torch.randn(2560)
target_basis = lens.transport(x, layer=16)
source_direction = lens.pullback(torch.randn(2560), layer=16, normalize=True)

See examples/basic_usage.py for a complete local example.

Limitations

• Use the exact model revision in provenance.json; maps cover only source layers 0–29 and target layer 30.
• Each map is a global average across prompts, positions, and future distances, so prompt-specific structure can be lost.
• Token readouts depend on the unembedding and can be distorted by rare or weakly trained vocabulary rows.
• Better transport metrics do not by themselves prove causal faithfulness.
• Mean-Future was tested on two Qwen3.5 sizes, not other model families.
• The dense artifact is about 750 MiB and loads into CPU memory.

Reproducibility

The raw fit prompts, exact sampled CKA token IDs, and internal operational records are not distributed.

License

Apache-2.0. The base model and datasets retain their own attribution and license requirements.