Downloads · 30 days
7
21% of all-time downloads
ceselder/skip-lens-qwen36-27b-fineweb-opd-scaled
skip-lens-qwen36-27b-fineweb-opd-scaled is a text generation model from ceselder. Use it when you need the model to write or continue text. It is set up for peft.
This repository contains the selected rank-64 rsLoRA adapter from a scaled activation-only future-lens experiment. The student receives only one residual-stream activation, injected at the Karvonen token; it cannot at…
Downloads · 30 days
7
21% of all-time downloads
All-time downloads
33
Public
Repo size
653 MB
Likes
0
Public
Click a slice to open those files.
.safetensors633 MB · 96%
From the Hugging Face model README
This repository contains the selected rank-64 rsLoRA adapter from a scaled activation-only future-lens experiment. The student receives only one residual-stream activation, injected at the Karvonen token; it cannot attend to the source text. A frozen Qwen3.6-27B teacher receives the actual FineWeb prefix and scores the student's own sampled continuation.
This is a research checkpoint, not a normal chat model. Loading it requires the custom activation-injection hook in skip-lens.
Qwen/Qwen3.6-27BFor OPD, the student samples exactly eight tokens at temperature 1.0. EOS remains in the sampled distribution but is disabled as a stopping condition, so the OPD and SFT arms are exactly token-matched. The teacher scores every student-sampled token on the real prefix plus the sampled prefix. Training uses the sampled per-token reverse-KL policy-gradient estimator with stored behavior log-probabilities and importance ratios, following Thinking Machines' on-policy distillation recipe.
The uploaded checkpoint is update 500, selected for minimum teacher-forced L62 validation NLL over checkpoints saved every 50 updates. Its comparison control is SFT at the same update and exact token budget.
| Fed activation | Arm | Reference NLL ↓ | Exact reverse KL ↓ | Teacher top-1 ↑ | Coherence /5 ↑ | Hallucination ↓ |
|---|---|---|---|---|---|---|
| L62 | OPD | 3.012 | 0.704 | 66.2% | 3.99 | 24.0% |
| L62 | Matched SFT | 2.945 | 0.737 | 66.6% | 4.11 | 31.2% |
| L42 | OPD | 3.451 | 1.126 | 57.9% | 3.78 | 27.7% |
| L42 | Matched SFT | 3.386 | 1.242 | 58.0% | 3.80 | 33.6% |
Paired L62 differences (OPD − matched SFT):
The evals/ directory contains the complete automatic analysis, raw selected-checkpoint
L62/L42 evaluations, and Sonnet-5 judgments.
adapter_model.safetensors, adapter_config.json: PEFT adapternla_meta.yaml: injection token, neighbor-token, layer, and prompt metadataopd_meta.json: training budget and stopping metadataselection.json: selected update, checkpoint paths, and selection criterionevals/: raw and summarized evaluation artifactsRepository: ceselder/skip-lens-qwen36-27b-fineweb-opd-scaled.