Downloads · 30 days
0
nla-qwen36-27b-matryoshka
ceselder/nla-qwen36-27b-matryoshka
nla-qwen36-27b-matryoshka is a machine learning model from ceselder. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
A matryoshka NLA trained on Qwen3.6-27B residual-stream activations (layer 42 of 64, d=5120): the activation verbalizer (AV) is RL-trained with random-length truncation of its explanation before reconstruction (U[1,12…
Access
Public
Updated Jul 14, 2026
Repo size
113 GB
Likes
2
Trending 1
Repo makeup
Click a slice to open those files.
.safetensors115 GB · 100%
At a glance
- License
- mit
- Access
- Public
- Created
- Jul 8, 2026
- Updated
- Jul 14, 2026
- SHA
- ad65f3db
Base models
Model card
From the Hugging Face model README
Matryoshka Natural Language Autoencoder — Qwen3.6-27B (layer 42)
A matryoshka NLA trained on Qwen3.6-27B residual-stream activations (layer 42 of 64, d=5120): the activation verbalizer (AV) is RL-trained with random-length truncation of its explanation before reconstruction (U[1,120] content tokens, shared per GRPO group), so the most important information is pushed to the front of every explanation.
Built with EasyNLA (branch qwen36-matryoshka),
following the NLA recipe of Anthropic (2026)
with the matryoshka truncation reward + position-tapered KL.
Results (400 GRPO steps, held-out)
| prefix seen by reconstructor | FVE |
|---|---|
| first 10 tokens | 42.9% |
| first 20 tokens | 51.9% |
| first 40 tokens | 58.0% |
| first 80 tokens | 61.5% |
| full explanation | 64.2% |
The first 10 tokens carry 67% of the full-explanation FVE — explanations open with the single most predictive fact (usually the immediate next-token completion) and broaden from there.
Contents
| path | what |
|---|---|
rl_av_lora_iter400/ | the matryoshka AV — LoRA (r128, attn + DeltaNet projections) on the text-only base, RL step 400 |
rl_critic_step400/ | co-trained AR reconstructor (43-layer truncated backbone + value head) at step 400 |
warmstart_av_lora/ | AV after SFT warm-start only (pre-RL baseline) |
warmstart_ar_critic/ | AR after SFT warm-start (48.2% held-out FVE on gold explanations) |
Note: LoRAs apply to a text-only conversion of Qwen3.6-27B
(Qwen3_5ForCausalLM: wrapper keys model.language_model.* remapped to model.*,
vision tower + MTP head dropped — bit-identical text logits). The AR critic dirs are
self-contained (truncated config + value_head.safetensors), loadable with EasyNLA's
NLACriticModel.from_pretrained.
Injection: marker ㈜ (id 158983), Karvonen norm-matched addition at layer-1 output.
Prompt format: bullets (untagged newline list), enable_thinking=False.
Training data: ceselder/nla-qwen36-27b-matryoshka-data.
- License
- mit
- Created
- Jul 8, 2026
- Updated
- Jul 14, 2026