vibethinker-3b-jlens-model

VibeThinker-3B J Lens

The J Lens is a Jacobian lens fitted to WeiboAI/VibeThinker-3B. Pick one of the 18 fitted source layers and a token position, and the lens decodes that residual-stream activation into a ranked list of vocabulary tokens. Following one position across layers shows how the decoded ranking changes on the way to the model's final output.

The lens contains 18 matrices, one for each even-numbered source layer from 0 through 34. Each matrix maps its source-layer residual-stream activation into layer-35 coordinates. VibeThinker-3B's final normalization and vocabulary projection produce the ranked token scores.

This repository contains the fitted lens in two Safetensors precisions. model.safetensors is the FP16 lens used to capture the released traces. evaluation.safetensors is the FP32 lens used for the recorded readout evaluation. Casting each FP32 matrix to FP16 reproduces the FP16 file exactly; both are included so every published result stays paired with the tensor values that produced it.

The companion trace repository contains six saved traces. Each trace pairs a prompt and its generated response with the top 12 decoded tokens at each captured position for the 18 source layers and the final model layer.

Choose a path

Companion repositories

Artifact identity

Both artifacts contain 18 matrices with shape 2,048 × 2,048. Their source layers are every even layer from 0 through 34, and their target layer is 35. The lens was fit on 1,000 prompts for model revision 77bd2cced09193c8b9a59a32bd8577bbd1f3e01c. The revision binding was reconstructed from the Hub head after the fit; provenance.json records that binding and its limits.

Files

• model.safetensors — the 18 losslessly reserialized FP16 lens matrices.
• evaluation.safetensors — the 18 losslessly reserialized FP32 evaluation matrices.
• lens_config.json — model binding, layer layout, fit parameters, and artifact ID.
• tensor_manifest.json — shape, dtype, size, and raw-byte SHA-256 for every tensor.
• evaluation_tensor_manifest.json — the corresponding FP32 tensor manifest.
• evaluation_provenance.json — derivation and source-hash bindings.
• evaluation_compatibility.json — the FP32-to-FP16 comparison for every layer.
• evaluation_validation.json — the exact FP32 conversion checks.
• provenance.json — fit, corpus, software, and conversion provenance.
• validation.json — exhaustive source-to-safetensors comparison result.
• evaluation.json — recorded readout-only evaluation and its scope.
• SHA256SUMS — checksums for both Safetensors files.
• requirements.txt — pinned packages for loading and validation.
• scripts/validate_artifact.py — validates the artifact without pickle.
• scripts/convert_checkpoint.py — reproducible checkpoint conversion for a user-provided source file.
• scripts/convert_evaluation_checkpoint.py — reproducible FP32 conversion.
• LICENSES/QWEN-RESEARCH.txt — complete distribution terms for the artifacts.
• NOTICE and THIRD_PARTY_NOTICES.md — required attribution and license context.

Tensor layout

The keys are J.<source_layer>:

J.0, J.2, J.4, J.6, J.8, J.10, J.12, J.14, J.16,
J.18, J.20, J.22, J.24, J.26, J.28, J.30, J.32, J.34

Each matrix maps the post-transformer-block residual at its source layer into the layer-35 residual basis. With row-major batches of residual vectors, the transport used by the reference implementation is:

transported = residual.float() @ J.float().T
logits = unembed(transported)

Both Safetensors headers contain the model ID and revision, source and target layers, width, prompt count, source-checkpoint hash, and key pattern.

requirements.txt pins the runtime used to create and validate evaluation.safetensors. provenance.json records the earlier runtime used to convert model.safetensors.

Install and load

The jlens Python package comes from the companion source repository. This repository contains the fitted tensors and the dependencies needed to inspect and validate them.

From this repository checkout, install both parts:

export JLENS_CODE_REPO_URL="https://github.com/jvogan/vibethinker-3b-jlens"
git clone "$JLENS_CODE_REPO_URL" ../vibethinker-3b-jlens
python3 -m pip install -r requirements.txt
python3 -m pip install ../vibethinker-3b-jlens

Load the local artifact:

from jlens import JacobianLens

lens = JacobianLens.load("model.safetensors")
print(lens.source_layers)

Or load an immutable Hugging Face revision, setting JLENS_MODEL_REVISION to the model-repository commit recorded in the source repository's release-manifest.json:

export JLENS_MODEL_REPO_ID="JacobMolBio/vibethinker-3b-jlens-model"
export JLENS_MODEL_REVISION="<commit from the source repository's release-manifest.json>"

import os

from jlens import JacobianLens

lens = JacobianLens.from_pretrained(
    os.environ["JLENS_MODEL_REPO_ID"],
    filename="model.safetensors",
    revision=os.environ["JLENS_MODEL_REVISION"],
)

To inspect tensors without the companion package, install requirements.txt and use Safetensors directly:

from safetensors import safe_open

with safe_open("model.safetensors", framework="pt", device="cpu") as handle:
    metadata = handle.metadata()
    jacobians = {
        int(key.split(".", 1)[1]): handle.get_tensor(key).clone()
        for key in handle.keys()
    }

assert metadata["model_revision"] == (
    "77bd2cced09193c8b9a59a32bd8577bbd1f3e01c"
)
assert sorted(jacobians) == list(range(0, 36, 2))

Load VibeThinker itself from the upstream repository at the exact bound revision; this repository does not include VibeThinker or Qwen weights:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "WeiboAI/VibeThinker-3B"
revision = "77bd2cced09193c8b9a59a32bd8577bbd1f3e01c"
tokenizer = AutoTokenizer.from_pretrained(model_id, revision=revision)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    revision=revision,
    dtype="auto",
)

Review and accept the upstream terms before downloading or using the model.

Viewer and new-prompt compute

The companion Pages site is a static viewer for captured trace JSON. It parses, filters, and renders those records in the browser; loading a local JSON file uses the browser's file reader and does not upload the file. The static site cannot analyze a new prompt. That requires the companion source package, this lens artifact, the pinned VibeThinker weights, and local Python compute.

Fit method

The lens was fit with the Apache-2.0 Anthropic Jacobian Lens reference implementation at commit 581d398613e5602a5af361e1c34d3a92ea82ba8e. For each fitted source layer, the estimator sums cotangents over valid causal targets at or after a source position, averages over valid source positions, and gives each prompt equal weight.

• Hook point: post-transformer-block output residual
• Source layers: every even layer from 0 through 34
• Target layer: 35
• Sequence cap: 128 tokens
• Leading positions skipped: 16
• Final position excluded from fitting
• Dimension batch: 8
• Fit runtime dtype: BF16
• Trace artifact dtype: FP16
• Evaluation artifact dtype: FP32
• Fit manifest: 1,000 examples from WikiText-103 raw train, minimum 600 characters

provenance.json records the fit manifest, a later local rematerialization of the prompt set, and the hash checks that bind them. Prompt text is not included in this repository.

Validation

Validation checked the following for every tensor in both artifacts:

• exact key set;
• exact shape and each artifact's declared FP16 or FP32 dtype;
• exact numerical equality with the recorded source checkpoint;
• exact FP16 or FP32 bit-pattern equality;
• exact raw tensor-byte SHA-256;
• contiguity and full finiteness; and
• a Safetensors header free of private source paths.

All 18 tensors in each artifact passed, and zero values changed. The FP32-to-FP16 cast also matches model.safetensors exactly for all 18 layers. Recheck both artifacts with:

python3 scripts/validate_artifact.py

The validator verifies both file checksums, cross-file model and artifact bindings, header metadata, tensor shapes and hashes, license and notice files, and FP32-to-FP16 compatibility.

Evaluation

The recorded evaluation measures ranked-token readouts: for each prompt in a 551-item suite, does a known target term rank highly in the decoded tokens? The paired token-target and shuffled-layer mapping checks passed for the selected band at layers 24, 26, 28, 30, 32, and 34. The comparison with the ordinary logit lens did not establish improvement. evaluation.json records this readout-only scope as validated_readout_only.

The 551 items cover association, multihop, multilingual, order-of-operations, poetry, and typo prompts, each pairing a prompt with one or more target terms. A deterministic split assigned 377 eligible items to the test set; the band was selected on development items. The companion trace repository contains all evaluation inputs, the 50,050 readout rows, the aggregate metrics, the bootstrap intervals, and the exact method, and its scripts/recalculate_readout.py recomputes every recorded statistic from the released rows. Regenerating the rows themselves requires the pinned base model, evaluation.safetensors, and a separate evaluator implementation.

The recorded metrics bind to evaluation.safetensors (SHA-256 0cc184eb65d273ac8bfee5450a141c8cdf8dd6ec8caa68d7d47260ef621777d1, from source FP32 checkpoint SHA-256 8f752032a26a5196c1cb447ef63f01e8a29820ff0c57178dd80c6e26d32b12a9); they do not evaluate model.safetensors. The evaluation does not measure general model accuracy or test causal steering or free-generation behavior.

The lens is stride-2, was fit on 1,000 prompts, and is model-revision specific. Single-cell readouts can be noisy; compare patterns across nearby layers and positions before interpreting an isolated token.

License and attribution

The Qwen Research License applies to this distribution and limits use to non-commercial research and evaluation. Commercial use requires a separate license from the upstream rights holder. Redistribution must include the complete Qwen Research License and retain the required notice in NOTICE. Built with Qwen. No VibeThinker or Qwen base-model weights are redistributed here.

The pinned VibeThinker model-card metadata declares license: mit and names Qwen/Qwen2.5-Coder-3B as its base model; the pinned tree contains no standalone license file. LICENSES/VIBETHINKER-LICENSE-NOTE.txt records that metadata and the pinned revision URL.

The fitting implementation is Anthropic's Apache-2.0 jacobian-lens. WikiText-103 is available under CC BY-SA 4.0; no source text is redistributed here. See THIRD_PARTY_NOTICES.md for details and links.

References

• Anthropic, Verbalizable Representations Form a Global Workspace in Language Models
• Anthropic, jacobian-lens
• WeiboAI/VibeThinker-3B
• Qwen/Qwen2.5-Coder-3B
• Salesforce/wikitext