qwen3.6-35B-A3B-av-sft

qwen3.6-35B-A3B-av-sft

Activation Verbalizer (AV) from a Natural Language Autoencoder (NLA) pair for Qwen/Qwen3.6-35B-A3B, trained with the open-source NLA pipeline. Companion model: qwen3.6-35B-A3B-ar-sft.

THIS IS THE SFT CHECKPOINT. FOR THE RL'D MODEL, SEE qwen3.6-35B-A3B-av-RL1200.

Given a layer-29 residual-stream activation (d=2048) injected at a marker token in its prompt, this model generates a natural-language description of the context that produced that activation.

Architecture

• Full Qwen3.6-35B-A3B (text side), LoRA r=32 (attention, GDN, shared+routed experts) merged into plain HF weights
• Injection: the activation vector, L2-normalized to injection_scale (see nla_meta.yaml / injection_scale.txt), replaces the embedding of the marker token ㊜-style char (injection_char in the sidecar) via a forward hook on the input embeddings — positions found by scanning input ids inside the hook

Training

• 2900 steps, global batch 256, lr 1e-4, single pass over ~742k unique (activation, summary) pairs — no repeats; CE 2.33 → 1.22
• Targets: conceptual summaries of the activation's source context, generated by Qwen3.6-35B-A3B itself, each ending with a description of what the final portion of the context is doing

Held-out roundtrip (AV → text → AR → vector, n=128)

Usage notes

nla_meta.yaml (included) carries the token ids, prompt templates, and scales. Injection must scan for the marker token id at inference time, not use precomputed positions. See the NLA repo for the injection utilities.