nla-qwen2.5-7b-universal-av-grpo

NLA Activation Verbalizer — Qwen 2.5 7B, universal, GRPO-refined

The universal SFT verbalizer sharpened with AR-native GRPO: the reward is whether an independent Activation Reconstructor can rebuild the original activation from the description alone. No semantic similarity model anywhere in the reward — faithfulness is scored in the model's own 3584-dim activation space.

Part of the nla-at-home project.

What GRPO changed (eyeball version)

SFT describes the genre of the activation; GRPO quotes its content. Same activation, layer 20:

SFT: "TypeError: Cannot read property" as a JavaScript runtime error · "Cannot read property" as a key-access failure pattern

GRPO: Error diagnosis and debugging assistance: "why does this code keep giving me an error" with "react hooks, usestate, out of bounds" · Response strategy: structured diagnostic response

Reward design (what made it work after three failed attempts)

reward = centered_cos(AR(desc), act) × specificity(‖AR(desc) − μ_L‖)

• centered cosine: per-layer mean removed from both sides — raw cosine is ~90% shared layer offset and rewards nothing.
• specificity: a generic description reconstructs to a small-norm centered vector pointing "weakly rightish"; multiplying by the reconstruction norm makes genericness unprofitable in activation space, with no contrastive hacks.
• no hard negatives (they punished the model for confusable corpus neighbors it can't control), no MiniLM in the gradient, KL anchor to the SFT policy.
• curriculum, not reward: an oracle compass (ridge map activation→text embedding) selects which (layer, text) pairs are decodable enough to train on, tau 0.40→0.10 over 8 epochs. The compass never touches the reward or the eval.

Best checkpoint: epoch 5/8, training reward 0.6747. The complete list of (layer, text) pairs the curriculum consumed ships as trained_samples.jsonl.

Evaluation — round-trip on a clean holdout

286 texts that none of AV training, AR training, or the GRPO curriculum ever touched (eval_holdout_ids.json, construction documented in eval_holdout_README.md). Descriptions generated greedy, reconstructed by the frozen AR, centered cosine:

GRPO wins at every depth. (For calibration: Anthropic's kitft L20 AV round-trips at 0.769 through their own AR.)

The confabulation tail — dissolved, not hidden

The fraction of confidently-wrong descriptions (low round-trip cosine) drops ~7×:

Forensics on the residual GRPO tail (per-item records ship with this repo): most low-cos items are correct descriptions of texts whose near-output (L27) geometry the AR reconstructs poorly — not hallucinations. Genuine content confusions exist but are rare (~1 in the worst-5). Raw per-item data: roundtrip_eval.records.jsonl.

Injection protocol

Identical to the SFT card: ㈎ (id 149705), normalize TO L2 norm 150.0, depth-conditioned prompt, chat template. GRPO-sharpened adapters are numerically touchy: use CUDA (bf16/fp16) or CPU fp32 — not MPS bf16.

Limitations

Same corpus scope as the SFT stage (safety-filtered public split). GRPO improves content specificity; it does not add knowledge of activation regions the corpus never covered.