ANDREA-120M-jlens

ANDREA-120M — BLAH Jacobian Lens Bundle

A blah-jacobian-lens (format version 1) bundle for russellbal/ANDREA-120M, consumed by evals.blah.dev to render layer-by-token Logit Lens and Jacobian Lens readouts.

This artifact is a readout instrument: fitted transport matrices that let internal representations be decoded through the model's own final decoding path, per site and per position. It is not a sparse autoencoder, and it makes no architectural claims (no workspace, broadcast, or causal-necessity claims) — those would require interventions and controls that are downstream work.

Base model

Exact file fingerprints (weights, config, tokenizer, chat template) are in lens-manifest.json; a consumer must refuse to apply this lens to any model whose fingerprints differ.

Sites

13 token-aligned sites of width 768, layout BTD:

Target site: block.011.post — the exact tensor consumed by the final decoding path. ANDREA applies no final norm, so exact_final_decode(h) = lm_head(h) with a float32 cast, and a direct Logit Lens is well-defined at every site (logit_lens_supported: true).

Lens

• Estimator kind: same_position — J_s = E_t[∂ target[t] / ∂ site_s[t]], cotangent injected at one target position at a time, gradient read at the same source position; positions [skip_first, T-2]; averaged over positions then prompts; float64 accumulation; dimension-batched VJP over (position, dimension) pairs.

• Application is affine (lens.centering.mode = "affine"):

  transported = target_mean + (h_s − source_mean_s) @ Jᵀ_s
  jacobian_logits = exact_final_decode(transported)
  

  Per-site means (mean.<site>, float32) and the target mean (mean.target) are stored in transports.safetensors alongside the matrices.

• Transports: dense, transport.0000–transport.0012, each [768, 768], stored J[out, in], applied h @ Jᵀ. Published dtype float16 (means float32), exported only after parity measurement against the float32 master — see exported_dtype_parity in validation.json.

Fitting corpus

Prompts sampled deterministically (seeded reservoir, seed 42) from the public russellbal/smol-seeds dataset at a pinned revision — an interleave of chat.txt and smoltalk.txt, two core sources of the megachat composite this model trained on. Fully public provenance: fit-report.json records the dataset revision, corpus fingerprint, filtering rules, counts, device, dtype, and elapsed statistics. Fitting parameters: see lens-manifest.json lens block.

Validation

validation.json carries the full suite: fingerprints, tokenizer parity, final-logit parity (exact to 0.0 by construction — the decode path is the model's own lm_head), determinism, VJP finite-difference check, matrix orientation, transport shapes, final-site identity, exported-dtype parity, split-half convergence, and golden-fixture reproduction. fixtures/golden.json + fixtures/golden.safetensors pin one raw prompt and one chat-formatted example with exact token ids, byte-exact token identities (bytes_base64 — Harris token strings use the GPT-2 byte-display mapping, so text alone is not byte-exact), captured activations, and top-k readouts.

Execution

execution.mode = "huggingface": evals.blah.dev workers load the base model from its repository at the pinned revision (custom code required, no auth). There is no public runtime URL; blah-lens-http/1 is not served.

Reproduction

From the model's source repository (training.ai.unturf.com, directory microgpt/):

# fetch chat.txt + smoltalk.txt from russellbal/smol-seeds at the revision in
# fit-report.json, then: paste -d'\n' chat.txt smoltalk.txt | grep -v '^$' > mix.txt
python3 scripts/jlens-corpus.py --dataset mix.txt \
    --samples 256 --heldout 8 --seed 42 --out corpus.jsonl \
    --corpus-name 'smol-seeds public chat+smoltalk interleave' \
    --identifier 'russellbal/smol-seeds (chat.txt + smoltalk.txt)' \
    --visibility public
python3 -m jlens fit --checkpoint <hf-snapshot-dir> --prompts corpus.jsonl \
    --samples 256 --max-seq-len 128 --skip-first 16 --dim-batch 96 \
    --estimator-kind same_position --dtype float32 --output dist/blah-lens
python3 -m jlens finalize --checkpoint <hf-snapshot-dir> \
    --repo-id russellbal/ANDREA-120M \
    --revision 6f81f00f9669db77d9fe6e62b525fac0650b34d8 \
    --corpus-meta corpus-meta.json --heldout corpus-heldout.jsonl \
    --output dist/blah-lens
npx @blahai/lens validate ./dist/blah-lens

Known limitations

• Fitted at max_seq_len 128 of a 1024 learned-position window; transports average position effects only over fitted positions.
• Fitting-corpus lines cap at 256 characters (a build-time cap in the seed datasets), so fitted sequences are short; 256 prompts compensate.
• The chat template has no system role — ANDREA trained without one.
• Readout instrument only: steering, suppression, ablation, and concept swaps are not part of this bundle.

License

AGPL-3.0, matching the base model. The fitting corpus is public (russellbal/smol-seeds, same project, same license family).