tofu-llama-3.1-8b-jlens-wikitext

J-Lens for TOFU Llama-3.1-8B-Instruct Full

This repository contains a fitted Jacobian Lens (J-Lens) for:

open-unlearning/tofu_Llama-3.1-8B-Instruct_full

The lens was fitted for interpretability experiments on intermediate representations of the TOFU fine-tuned Llama-3.1-8B-Instruct model.

Fitting Data

The J-Lens was fitted using:

Salesforce/wikitext

WikiText was used instead of the TOFU evaluation data to avoid fitting the lens specifically to the evaluation examples.

A total of 1,000 prompts were requested, and all 1,000 prompts were successfully used for fitting.

Fitting Configuration

• Base model: open-unlearning/tofu_Llama-3.1-8B-Instruct_full
• Fitting corpus: Salesforce/wikitext
• Requested prompts: 1,000
• Successful fitting prompts: 1,000
• Source layers: 0–30
• Target layer: 31 (final transformer layer)
• Maximum sequence length: 128
• Skip first tokens: 16
• Dimension batch size: 4
• Random seed: 20260714

The source layers follow the default fitting configuration of the reference J-Lens implementation. For a 32-layer model, the final transformer layer (layer 31) is used as the target layer, and all preceding layers (0–30) are used as source layers.

Files

lens.pt

The final fitted Jacobian Lens.

This is the main file intended for downstream J-Lens inference and interpretability experiments.

lens.pt.ckpt

A resumable checkpoint from the J-Lens fitting process.

It stores the accumulated fitting state and can be used to resume or extend fitting. This file is not required when only using the final fitted lens for analysis.

lens.pt.metadata.json

Metadata describing the model, fitting corpus, source layers, and fitting configuration used to generate the lens.

Usage

With the reference jlens implementation installed, the fitted lens can be loaded directly from this Hugging Face repository:

from jlens import JacobianLens

lens = JacobianLens.from_pretrained(
    "da3eun/tofu-llama-3.1-8b-jlens-wikitext"
)

print(lens)

Only lens.pt is downloaded when loading the lens with JacobianLens.from_pretrained() using its default filename.

Method

The Jacobian Lens reads out an intermediate residual-stream representation by linearly transporting it into the final-layer representation space using an average input-output Jacobian and then decoding it with the model's own unembedding.

For each fitted source layer (l), the lens contains an averaged Jacobian matrix (J_l) that maps the residual representation at layer (l) into the final-layer basis.

The methodology and implementation follow the Jacobian Lens introduced in:

Verbalizable Representations Form a Global Workspace in Language Models

Reference implementation:

anthropics/jacobian-lens

Intended Use

This artifact is intended for research on language-model interpretability, particularly for analyzing intermediate hidden representations of:

open-unlearning/tofu_Llama-3.1-8B-Instruct_full

The lens was fitted for experiments involving intermediate-representation decoding and comparison with other lens-based analysis methods.

This repository contains J-Lens artifacts only and does not redistribute the underlying Llama or TOFU model weights.

Notes

• The lens was fitted on WikiText rather than the TOFU evaluation set.
• The lens is model-specific and is intended for use with open-unlearning/tofu_Llama-3.1-8B-Instruct_full.
• lens.pt is sufficient for standard J-Lens analysis.
• lens.pt.ckpt is included for reproducibility and for resuming the fitting process.
• Layer 31 is the target layer and therefore is not included among the fitted source layers.
• The fitting metadata is also provided separately in lens.pt.metadata.json.

Attribution and Licenses

The Jacobian Lens reference implementation is released under the Apache License 2.0.

The fitting corpus, Salesforce/wikitext, is distributed under its applicable WikiText licensing terms.

The base model used to fit this artifact is based on Meta Llama 3.1 and is subject to the applicable licensing and usage conditions of the corresponding model and Llama 3.1.

Built with Llama.

Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved.

Users of this artifact are responsible for reviewing and complying with the licenses and usage conditions of the underlying model, dataset, and reference implementation.