Qwen3-14B-Per-Layer-Refusal-Directions-alpha-1

Qwen3-14B Per-Layer Refusal-Direction Edit (alpha=1)

Experimental research condition, released for comparison and reuse. This checkpoint is not the preferred result of the associated study. It is published so that researchers can inspect, reproduce, and extend the tested per-layer intervention.

Model summary

For layer-specific unit direction r_l, each supported output-space weight in layer l is edited as:

W_l,new = W_l - r_l (r_l^T W_l)

Each block uses its original resid_post[l] direction for its attention o_proj and MLP down_proj. The untied embedding uses the block-0 output direction. The unembedding, Q/K/V, gate/up projections, and RMSNorm tensors are unchanged.

Evaluation

All comparisons use fixed prompts and matched decoding settings. Prompt and completion bodies are not distributed with this model.

An independent garak-llm/garak-refusal-detector classifier measured 51.46% refusal for Base and 0.00% for this checkpoint on the same 410-sample refusal set. It is a binary synthetic-data classifier, not the official StrongREJECT evaluator, and does not establish response correctness or safety.

On complete IFEval (541 prompts, 834 instructions), this checkpoint scored 86.32% prompt-strict, 90.53% instruction-strict, 88.54% prompt-loose, and 91.97% instruction-loose.

Interpretation and known limitations

This experiment does not show that Per-layer editing is better than the simpler Global method. At alpha=1, refusal reduction is similar while Per-layer has roughly three times the benign KL and a 1.0-point lower capability macro score. The paper's main empirical conclusion therefore does not favor this checkpoint.

The initial refusal evaluation reached its 256-token cap for 94.39% of Per-layer responses. Unlike Global alpha=0.75, this checkpoint did not receive a dedicated 1024-token output-degradation and relevance rerun. Its long-generation stopping, repetition, and off-topic rates are therefore uncharacterized, not assumed to be acceptable.

Per-layer directions are statistically stable under 200 bootstrap and 200 half-sample recomputations, and their pairwise cosine geometry rotates across model depth. Stable geometry does not by itself imply a superior intervention trade-off.

Intended use and limitations

This checkpoint is intended for mechanistic-interpretability, model-editing, and controlled evaluation research. It is not presented as safe, universally "uncensored," capability-preserving for every task, or suitable for deployment. Reducing refusal behavior does not show that the model has lost its internal representation of harmfulness.

No public API, Space, managed endpoint, or interactive service is maintained by this project.

Reproducibility

Code, configurations, aggregate results, plots, and the technical report are available at VecToR-1024/qwen3-refusal-direction-removal. The repository excludes extraction prompt bodies and restricted generations.

The serialized checkpoint passed save/reload, six-shard index, activation-versus-weight, determinism, dtype/device, and tied-weight handling checks before release. Users should perform a fresh-download load and benign smoke test against the exact Hub revision they consume.

License

The base model and this derivative checkpoint are distributed under Apache-2.0. The base model's notices and attribution remain applicable. Dataset licenses remain applicable to their source data; dataset bodies are not redistributed here.