Downloads · 30 days
39
13% of all-time downloads
gjtgjt/Qwen3.8-27B-heretic-r1n
Qwen3.8-27B-heretic-r1n is a image-text-to-text model from gjtgjt. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.
Iterative true rank-1 (PRE) Heretic of Qwen/Qwen3.8-27B. After each accepted round the residual direction r is recomputed on the updated weights, then Heretic is run again. Cumulative KL vs the original model is cappe…
Downloads · 30 days
39
13% of all-time downloads
All-time downloads
308
Public
Parameters
27.4B
54.7 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors54.7 GB · 100%
From the Hugging Face model README
Iterative true rank-1 (PRE) Heretic of Qwen/Qwen3.8-27B. After each accepted round the residual direction r is recomputed on the updated weights, then Heretic is run again. Cumulative KL vs the original model is capped at 0.1.
对 Qwen/Qwen3.8-27B 做 真 rank-1(PRE) 迭代 Heretic:每轮接受后再在新权重上重算残差方向 r,然后进入下一轮。相对原模型的累计 KL 上限为 0.1。
| Base | Qwen/Qwen3.8-27B, revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 |
| Architecture | Qwen3.8 hybrid: 64 layers, 16 × (3 × Gated DeltaNet → FFN, 1 × Gated Attention → FFN) |
| Method | Heretic 1.4.0, --row-normalization PRE |
| Ablated modules | language-model attn.o_proj / linear-attn out_proj, and mlp.down_proj (64 layers) |
| Untouched | vision tower, MTP |
| Precision | BF16 merged weights, 12 shards, ~51 GB |
| Context | 262,144 tokens (same as base) |
| License | Apache-2.0 (inherited from the base) |
Heretic is training-free directional ablation (Arditi et al., 2024). It estimates a per-layer residual direction r from the first generated token, then writes a low-rank update into the output projections so that component is suppressed. Optuna TPE searches ablation weights to lower refusal-keyword hits while keeping KL to the parent model small.
Why PRE (true rank-1). Heretic row-normalization has three modes:
none: ΔW = -λ v (vᵀ W) — rank-1, no row-norm handlingpre: same rank-1 form, but computed on row-normalized W, then scaled back by the original row norms: ΔW = -λ v (vᵀ W_norm) ⊙ ‖W‖_rowfull: apply the PRE update, renormalize rows to restore magnitudes, then keep an SVD approximation of rank full_normalization_lora_rank (default 3). That is not rank-1.This release uses PRE only. Each accepted round is therefore one true rank-1 update. Two accepted rounds are two successive rank-1 updates, not a single rank-2 factorization.
Why iterate, and why recompute r. After a merge the residual geometry changes, so the next direction is measured on the new weights. Reusing r from round 1 would not be a fresh rank-1 step.
KL accounting. Search-time KL in a Heretic log is vs the current parent (incremental). Acceptance always re-evaluates vs the original base with --evaluate-model. Only that cumulative KL is constrained to ≤ 0.1.
Thinking-model cut. Qwen3.8 thinks by default. Residuals and keyword scoring are taken at the start of the visible answer, with --response-prefix $'\n</think>\n\n'. Do not put that prefix into the chat template at inference time.
Pipeline
Qwen/Qwen3.8-27B (not from a previous heretic checkpoint).r on the round-1 merge.r, try round 3 → reject (cumulative KL 0.1983 > 0.1).Heretic default English eval: keyword refusals / 100 prompts, and KL on the harmless split.
| Round | Keywords / 100 | KL vs original | Decision |
|---|---|---|---|
| Base | 98 | 0 | — |
| 1 (trial 162) | 25 | 0.0525 | accepted |
| 2 (trial 109) | 18 | 0.0931 | accepted, this release |
| 3 (Pareto 0/1/2) | 8 | 0.1983 | rejected (KL > 0.1) |
Round 3 would have lowered keywords further, but broke the KL cap. The shipped weights stay at round 2.
Per-layer directions. Ablation strength is a piecewise-linear kernel over layer index.
| Parameter | Value |
|---|---|
direction_index | per layer |
attn.o_proj.max_weight | 1.14 |
attn.o_proj.max_weight_position | 60.59 |
attn.o_proj.min_weight | 1.05 |
attn.o_proj.min_weight_distance | 28.96 |
mlp.down_proj.max_weight | 0.97 |
mlp.down_proj.max_weight_position | 55.65 |
mlp.down_proj.min_weight | 0.01 |
mlp.down_proj.min_weight_distance | 24.00 |
Round 1 (already baked in): trial 162, attn.o_proj max 0.90 @ 53.74, mlp.down_proj max 1.31 @ 62.63.
Needs a Transformers build that loads Qwen3.8 (qwen3_5 / Qwen3_5ForConditionalGeneration), e.g. Transformers 5.15+. Same chat template and sampling defaults as the base.
from transformers import AutoModelForImageTextToText, AutoTokenizer
model_id = "YOUR_USER/Qwen3.8-27B-heretic-r1n"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
model_id, dtype="auto", device_map="auto",
)
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain residual connections in one paragraph."},
]
prompt = tok.apply_chat_template(
messages,
add_generation_prompt=True,
tokenize=False,
enable_thinking=False, # True (default) keeps <think>…</think>
)
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(
**inputs,
max_new_tokens=512,
temperature=0.7,
top_p=0.8,
top_k=20,
presence_penalty=1.5,
)
print(tok.decode(out[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True))
enable_thinking=False, temperature=0.7, top_p=0.8, top_k=20, presence_penalty=1.5enable_thinking=True (default), temperature=1.0, top_p=0.95, top_k=20; optional reasoning_effort in {xhigh, medium, low}Exact trial indices depend on the Optuna study. The procedure is:
pip install -U heretic-llm
PREFIX=$'\n</think>\n\n'
heretic --model Qwen/Qwen3.8-27B \
--response-prefix "$PREFIX" --row-normalization PRE \
--n-trials 200 --n-startup-trials 60 \
--model-action save --export-strategy merge \
--save-directory ./r1
# recompute r on the new weights, then search again
heretic --model ./r1 \
--response-prefix "$PREFIX" --row-normalization PRE \
--n-trials 120 --n-startup-trials 40 \
--model-action save --export-strategy merge \
--save-directory ./r2
# cumulative KL must be scored against the original base
heretic --model Qwen/Qwen3.8-27B \
--evaluate-model ./r2 --response-prefix "$PREFIX"
在官方 Qwen3.8-27B 上用 Heretic 1.4.0 做迭代真 rank-1 消融。PRE 在行归一化后的 W 上写 ΔW = -λ v (vᵀ W_norm),再乘回行范数,每轮严格 rank-1。FULL 会再做行范数还原并用默认 rank-3 的 SVD 近似,因此不是本仓库的做法。
每轮接受后必须在新权重上重算残差方向 r,不能复用上一轮的 r。搜索日志里的 KL 是相对本轮父模型的增量;验收一律对原模型跑 --evaluate-model,累计 KL 不得超过 0.1。Qwen3.8 默认带思考,取点和打分都切在 \n</think>\n\n 之后,推理时不要把这个 prefix 写进模板。
交付点是第 2 轮 trial 109:Keyword 18 / 100,累计 KL 0.0931。第 3 轮 Keyword 可到 8,但 KL 0.1983 超标,未合并。视觉塔和 MTP 未改。权重为 BF16 merge,12 个分片。许可 Apache-2.0。