skiplens-ar-L62-allidx

skiplens-ar-L62-allidx — truncation-resistant activation reconstructor (AR), Qwen3.6-27B

LoRA (r64 / α16, rsLoRA, lora_scope=all) + Linear(5120,5120) value head on a 63-layer-truncated Qwen3.6-27B. Reconstructs the layer-62 residual-stream activation from a short text span (≤12 tokens), trained with the dense --ar-all-idx objective (reconstruct at every causal prefix → read-position robust). Used as the frozen reward in futurelens-NLA GRPO.

• Base: Qwen/Qwen3.6-27B (truncated to 63 layers, final RMSNorm stripped)
• Training: 776k on-policy ≤12-token spans, 1 epoch, lr 1e-4, batch 32
• Held-out FVE (norm-constrained): 24.9% averaged over 4–12-tok spans (~30% at full length); baseline (predict-the-mean) ≈ 0
• Read template: Summary of the following text: <text>{span}</text> <summary> (reads at the last token; but all-idx → robust at any read position)
• Files: ar_lora_value_head.safetensors (LoRA + value head), ar_meta.json, nla_meta.yaml
• ⚠ Loading gotcha: ar_meta.json["target_modules"] was regenerated to the true set, but if you hit a stale copy, derive target_modules from the .lora_ keys in the safetensors (full set = q/k/v/o_proj, in_proj_a/b, in_proj_qkv/z, out_proj, gate/up/down_proj).