Downloads · 30 days
26
21% of all-time downloads
Solshine/qwen3.5-4b-nla-L23-av-priordev-relabel-v2
qwen3.5-4b-nla-L23-av-priordev-relabel-v2 is a text generation model from Solshine. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as cc-by-4.0.
LoRA adapter for Qwen/Qwen3.5-4B that takes a 2560-dimensional residual-stream activation captured at layer 23 and emits a short natural-language string.
Downloads · 30 days
26
21% of all-time downloads
All-time downloads
122
Public
Repo size
650 MB
Likes
0
Public
Click a slice to open those files.
.safetensors649 MB · 100%
From the Hugging Face model README
LoRA adapter for Qwen/Qwen3.5-4B that takes a 2560-dimensional residual-stream activation
captured at layer 23 and emits a short natural-language string.
This adapter does not produce usable descriptions of the activations you give it. It is a methodology artifact, released so that a cross-architecture comparison is checkable. It is not a working interpretability tool and should not be used as one.
What the measurements support, stated exactly: a statistically significant pool-wide retrieval signal ($p=0.023$, single seed), which this program's own routing-versus-content decomposition attributes principally to coarse topic routing. The within-domain axis, the one that isolates content, is not significant ($p=0.117$). Exact-document top-1 is 3/294 against a 1/294 shuffle control, which is chance. The generated text is confabulated and frequently degenerates into repetition loops: see the verbatim samples below.
An earlier version of this card said the recipe "transfers" and that this adapter "reads content". Both overstate what was measured, and are withdrawn. What transfers is a routing signal whose size tracks how completely the recipe is followed.
This is a cross-architecture port of the recipe developed on Gemma-4-E2B
(gemma-4-e2b-nla-L23-av-priordev-relabel-v1-wd3),
and the question it answers is narrower than "does the recipe work elsewhere": it is whether
recipe fidelity or architecture governs the size of the effect that does appear.
Three consecutive rows from the held-out evaluation, by document id, with nothing selected for or against. The eval pool is deliberately out-of-domain: this adapter was trained on news.
| source document (first ~90 chars) | what the adapter wrote |
|---|---|
#Person1#: OK, Mrs Thomas. I finished cleaning up the leaves in the yard. | "WJZ Baltimore brief, HS 160 context: Foster as resident expert on coastal erosion investing in erosion-control rocks; quote chain continues to eroding recreational beaches.</explanation> Her..." |
I used to love D&B when it first opened in the Waterfront, but it has gone down hill | "Thomas & Mack Center's heightened security explained by Wednesday's third presidential debate; Menzies' 'logistical nightmare' quote sets up UNLV basketball displacement details..." |
#Person1#: Is anybody in? #Person2#: How can I help you? #Person1#: I have a headache. | "Wests Tigers columnist finished condemning Moses's mid-season exit; Moses is now offering tea to a distressed Stroud, and the 'led by Moses' line awaits completion.</concept> Tyler Adams tra..." |
The outputs are fluent, confident, shaped like the news corpus the adapter was trained on, and
unrelated to their inputs. They leak training-template tokens (</explanation>, </concept>)
and some rows repeat a phrase until the token budget runs out. The percentile effect below is
measured against this, not against something more readable.
This adapter is the third attempt at this port. The first two are informative and are documented here rather than quietly dropped, because the difference between them is the result.
| Level | What changed | Outcome on Qwen3.5-4B |
|---|---|---|
| L1 | Relabeled corpus + injection mechanism ported, but trained with plain cross-entropy (LoRA r=64/α=128, lr=1e-4, 300 steps) | Null. The shuffled-activation control beat real injection on 2 of 3 content metrics. Reproduces the known Gemma-4 failure mode for this same omission |
| L2 | Prior-deviation-weighted objective restored with Gemma's hyperparameters, but the objective's prior model was the raw untrained base model | Weak but real: full-pool percentile 0.4825, 1.75 points off chance |
| L3 (this release) | The prior model matched to Gemma's recipe: a confirmed injection-blind, content-free checkpoint instead of the untrained base | full-pool percentile 0.4468, 5.32 points off chance — roughly triple L2's effect |
The decisive ingredient at L1→L2 is the prior-deviation-weighted objective; at L2→L3 it is what that objective computes its prior against. Gemma's winning run conditions the prior on a previously-trained verbalizer; using the untrained base model instead is a weaker prior, and closing that one gap tripled the effect without changing anything else — same corpus, same hyperparameters, same 6200-step horizon.
294-row held-out pool. Lower percentile is better (0.5 = chance rank of the true source document among all candidates); higher top-1 is better. Every number below comes from the same eval run, with both controls.
| condition | full-pool percentile | same-domain percentile | top-1 | mean cos to own doc |
|---|---|---|---|---|
| real injection | 0.4468 | 0.4597 | 1.02% (3/294) | 0.0562 |
| shuffled activation | 0.4981 | 0.4986 | 0.34% (1/294) | 0.0428 |
| no injection | 0.5000 | 0.5000 | 0.34% (1/294) | 0.0318 |
Real injection beats both controls on every metric, and the controls sit essentially exactly at chance, which is what makes the real-injection departure readable.
What that does and does not mean. Full-pool percentile ranks the true document against the whole 294-document pool, so getting the broad topic neighbourhood right is enough to move it. Same-domain percentile ranks it only against others from its own domain, so it is the axis that isolates reading the specific document. The first is significant, the second is not. Under this program's own routing-versus-content decomposition, that is a routing signal with no demonstrated content signal underneath it. Top-1 of 3/294 against a 1/294 shuffle control is two extra hits and should be read as chance.
Injection-sanity gate: passed, perfectly. Real-injection unique-output ratio 1.000 (all 294 generations distinct) against 0.003 for the no-injection control, a 294× gap. This gate runs before any content number is read, because this project has hit silent injection failures three separate ways; a checkpoint that trains cleanly while reading nothing is the failure mode it exists to catch.
The same L3 recipe was applied to
lfm2.5-2.6b-nla-L21-av-priordev-relabel-v2.
On matched methodology, the smaller model carries the larger routing signal: LFM2.5-2.6B
reaches 11.11 points off chance on full-pool percentile against this adapter's 5.32. Neither
family is significant on the same-domain axis, so this compares routing, not content. Model
size did not predict which family showed the larger effect. This adapter's one advantage is
injection diversity (1.000 vs 0.939), which measures mechanism liveness, not correctness.
| Base model | Qwen/Qwen3.5-4B |
| Read/inject layer | 23 (a full-attention layer of this hybrid stack, not a linear-attention one) |
| d_model | 2560 |
| Marker token | § (single token, id 17317), embedding overwritten at the marker position |
| Objective | Prior-deviation-weighted cross-entropy, γ=1, prior = injection-blind relabel_v1 checkpoint |
| LoRA | r=80, α=160 |
| Optimizer | AdamW, lr 5e-5, weight decay 0.3, grad_accum 2 |
| Steps | 6200 (≈20.6 epochs on this 600-row corpus, epoch-matched to Gemma's horizon) |
| Precision | fp16 base + fp32 LoRA |
| Hardware | Kaggle free-tier GPU session |
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-4B", dtype="float16", device_map="auto")
model = PeftModel.from_pretrained(base, "Solshine/qwen3.5-4b-nla-L23-av-priordev-relabel-v2")
The activation is L2-normalised, rescaled, and overwrites the input embedding at the marker
token position. Reproducing the numbers above requires that injection mechanism, not just the
adapter; see inject_config.json in this repo for the exact constants, and run the
shuffled-activation and no-injection controls before trusting any generation-based number.
| Path | Contents |
|---|---|
adapter_model.safetensors, adapter_config.json | The LoRA adapter |
inject_config.json | Injection constants (marker token, layer, scale). Required to reproduce the numbers — the adapter alone is not enough |
nla_meta.yaml | Training metadata |
eval/ | The evaluation artifacts behind every number above: raw generations and scored metrics for the real, shuffled-activation and no-injection conditions |
figures/crossarch_recipe_fidelity.png | The recipe-fidelity ladder and cross-family comparison |
Every figure and table in this card is derived from the files in eval/, so the claims here
are checkable against their own artifacts rather than only reproducible in principle.
Part of the GPU-poor NLA program. Full methodology, the recipe-fidelity analysis summarised
above, and all controls are in the accompanying paper; the source repository is
SolshineCode/deception-nanochat-sae-research.