Downloads · 30 days
24
9% of all-time downloads
YFC-112358/Qwen3.6-27B-Della-Deckard-v1
Qwen3.6-27B-Della-Deckard-v1 is a image-text-to-text model from YFC-112358. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.
A two-stage weight merge on top of Qwen/Qwen3.6-27B: a DELLA merge of two same-generation reasoning-distill sources, plus a small cross-generation task vector carrying engineering ability from a Qwen3.5 fine-tune.
Downloads · 30 days
24
9% of all-time downloads
All-time downloads
270
Public
Parameters
27.8B
55.6 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors55.6 GB · 100%
From the Hugging Face model README
A two-stage weight merge on top of Qwen/Qwen3.6-27B: a DELLA merge of two same-generation reasoning-distill sources, plus a small cross-generation task vector carrying engineering ability from a Qwen3.5 fine-tune.
Every mixing coefficient in this card was picked from measured weight-space distances, not guessed. The measurements are published below so you can disagree with them.
Two deltas are pruned, sign-elected, and added to the base:
| Source | Type | weight | density | epsilon |
|---|---|---|---|---|
Ravionhf/qwen3.6-27b-reasoning-distill-lora-v1 | LoRA (r=32, α=64, rsLoRA off → scaling 2.0) | 0.5 | 0.5 | 0.1 |
nerkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill | full weights | 0.5 | 0.6 | 0.1 |
magprune — within each block, entries are ranked by magnitude and assigned a
keep probability p interpolated linearly over [density − ε, density + ε].
A Bernoulli mask is drawn and survivors are rescaled by 1/p, so the pruned
delta stays an unbiased estimator of the original.
Sign election — the two pruned deltas are stacked; the majority sign is taken per-element and any contribution disagreeing with it is dropped before summing. This is what keeps two independently-trained reasoning distills from cancelling each other out.
The LoRA covers all 64 layers (304 modules across in_proj_{a,b,z,qkv},
out_proj, q/k/v/o_proj). It does not touch the MTP head.
τ = W(nightmedia/Qwen3.5-27B-Engineer-Deckard-Gemini) − W(Qwen/Qwen3.5-27B)
W_final = W_stage1 + 0.15 · τ
This moves an engineering/coding task vector across a model generation. That is only legitimate if the two generations still share a loss basin — see the measurement below.
Median over 17 automatically-selected probe tensors (MLP down/gate/up,
linear_attn.in_proj_qkv at layers 0/20/42/63, plus embed_tokens), sampled
8 rows each over HTTP Range requests — about 4 MB of traffic total.
| # | Pair | cos | ‖Δ‖/‖W‖ |
|---|---|---|---|
| ① | Deckard vs Qwen3.5-27B | 0.9996 | 0.0276 |
| ② | Qwen3.5-27B vs Qwen3.6-27B | 0.9478 | 0.3208 |
| ③ | vision tower: Deckard vs Qwen3.5 | 1.0000 | 0.0018 |
| ④ | nerkyor vs Qwen3.6-27B | 0.9997 | 0.0266 |
| ⑤ | vision tower: nerkyor vs Qwen3.6 | 1.0000 | 0.0000 |
| ⑥ | LoRA ΔW vs Qwen3.6-27B | — | 0.0057 |
Row ② is the gate for Stage 2. cos = 0.9478 between the 3.5 and 3.6 base weights means the generational update was a large but directionally coherent move — neuron correspondence survived, so a task vector computed in 3.5 space still points somewhere meaningful in 3.6 space. If this had come back near zero cosine, Stage 2 would have been noise injection and was to be disabled.
‖Δ‖/‖W‖ = 0.3208 also sets the ceiling. λ₂ was held at 0.15 rather than
0.3 specifically because the generational drift is already a third of the
weight norm; a larger transplant risks pushing weights outside the basin.
Resulting contribution budget (relative to base weight norm):
| Contribution | Product | Share |
|---|---|---|
| nerkyor | 0.5 × 0.0266 | 0.0133 |
| Deckard τ | 0.15 × 0.0276 | 0.0041 |
| LoRA | 0.5 × 0.0057 | 0.0029 |
Stage 2 sits at 0.31 × the DELLA main ingredient — a supporting voice, not a second opinion loud enough to overwrite the merge. All three terms are within one order of magnitude of each other, which is the condition for sign election to be meaningful rather than a rubber stamp for the largest delta.
| Group | Tensors | Treatment |
|---|---|---|
Vision tower (model.visual.*) | 333 / 0.86 GB | Copied verbatim from base. Rows ③ and ⑤ measured every donor as bit-identical to base here, so there was nothing to merge. |
MTP head (mtp.*) | 1 layer | Excluded from Stage 2. The LoRA never targeted it. |
norm, embed_tokens, lm_head, a_log, dt_bias, conv1d | — | Merged, but not pruned. Magnitude pruning on normalization scales and gating parameters is destructive; these are low-parameter, high-leverage tensors. |
‖out‖/‖base‖ ∈ [0.95, 1.08]. All 26 passed.{BF16}.AutoConfig + AutoTokenizer resolve; 248,077-token vocab; chat template present.Sampling: temperature=0.6, top_p=0.95, top_k=20, max_tokens=8192.
Thinking budget is well-calibrated — the most common failure mode for merged
reasoning models is <think> spam on trivial input. It does not happen here:
| Prompt | think tokens |
|---|---|
你好 | 27 |
法国的首都是哪座城市? | 28 |
1+1等于几? | 43 |
2 的 3 次方是多少?只要答案,不要解释。 | 50 → answered 8, nothing else |
| CRT congruence problem | 356 |
| 5-person ranking puzzle | 573 |
sin20°·sin40°·sin80° exact value | 1022 |
Verifiable answers were correct — CRT problem → 23; ranking puzzle →
B, D, A, E, C; the trigonometric identity was attacked via the triple-angle
form sin3θ = 4 sinθ sin(60°−θ) sin(60°+θ), which is the intended route.
Instruction following held on the "answer only" prompt.
No embedding damage — three Chinese-only prompts (expository writing,
《吕氏春秋》 idiom exegesis, 陶渊明 classical-Chinese gloss) produced fluent output
with no code-switching, no garbled long-tail tokens, no repetition. This was the
specific risk being probed, since embed_tokens and lm_head participate in
the merge.
Code generation intact — LRUCache from scratch (hash map + doubly-linked
list, __slots__, O(1) both ops) and a GIL/race-condition diagnosis with a
threading.Lock fix.
from vllm import LLM, SamplingParams
from transformers import AutoTokenizer
M = "YFC-112358/Qwen3.6-27B-Della-Deckard-v1"
tok = AutoTokenizer.from_pretrained(M)
llm = LLM(model=M, max_model_len=16384, gpu_memory_utilization=0.90)
sp = SamplingParams(temperature=0.6, top_p=0.95, top_k=20, max_tokens=8192)
prompt = tok.apply_chat_template(
[{"role": "user", "content": "你好"}],
tokenize=False, add_generation_prompt=True,
)
print(llm.generate([prompt], sp)[0].outputs[0].text)
from transformers import AutoModelForCausalLM, AutoTokenizer
M = "YFC-112358/Qwen3.6-27B-Della-Deckard-v1"
tok = AutoTokenizer.from_pretrained(M)
model = AutoModelForCausalLM.from_pretrained(M, dtype="bfloat16", device_map="auto")
Notes
<think> block. Split on </think> to separate reasoning from the answer.Could not find nvcc, set VLLM_USE_FLASHINFER_SAMPLER=0 before importing vllm.λ₂ = 0) if behavior seems off.Built on a free-tier Colab CPU runtime (12.7 GB RAM, 87 GB disk, no GPU) by streaming tensors row-wise over HTTP Range requests, so peak memory stays flat regardless of tensor size and no donor is ever fully resident on disk. Total download ≈ 230 GB. The merge notebook is included in this repo.
DELLA: DARE the Extreme — Revisiting Delta-Parameter Pruning (Deep et al., 2024). Task arithmetic: Editing Models with Task Arithmetic (Ilharco et al., 2023).
在 Qwen/Qwen3.6-27B 上做的两阶段权重合并:同代的两个推理蒸馏源做 DELLA 融合, 再叠加一个来自 Qwen3.5 微调模型的小幅跨代任务向量,用于移植工程/代码能力。
所有配比系数都来自实测的权重空间距离,不是拍脑袋定的,测量结果全部公开在上面的表里。
Stage 1 — DELLA(λ₁ = 1.0)
| 来源 | 类型 | weight | density | epsilon |
|---|---|---|---|---|
Ravionhf/qwen3.6-27b-reasoning-distill-lora-v1 | LoRA r=32 α=64,scaling 2.0 | 0.5 | 0.5 | 0.1 |
nerkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill | 全量权重 | 0.5 | 0.6 | 0.1 |
magprune 按幅度排名给保留概率,再除以概率做无偏还原;两路 delta 经符号选举后相加, 与多数派符号相反的贡献直接丢弃——这是防止两个独立训练的推理蒸馏互相抵消的关键。
Stage 2 — 跨代任务向量(λ₂ = 0.15)
τ = W(Deckard-3.5) − W(Qwen3.5-27B)
W_final = W_stage1 + 0.15 · τ
关键是②:官方 3.5 与官方 3.6 之间 cos = 0.9478、‖Δ‖/‖W‖ = 0.3208。 余弦仍然很高,说明代际更新虽然幅度大,但方向是连贯的,神经元对应关系没有被打乱—— 这才使得「在 3.5 空间算出来的任务向量搬到 3.6 空间仍然指向有意义的方向」成立。 如果这一行的余弦接近 0,Stage 2 就等于注入噪声,应当直接关掉。
同时 0.3208 也定了上限:λ₂ 保守取 0.15 而不是 0.3,因为代际漂移本身已经占到权重范数的三分之一。
最终三项贡献量级:nerkyor 0.0133 / Deckard 0.0041 / LoRA 0.0029, Stage 2 是 DELLA 主料的 0.31 倍——配角而非主角,且三项在同一数量级内, 符号选举才有意义(否则就退化成给最大的那个 delta 盖章)。
norm / embed_tokens / lm_head / a_log / dt_bias / conv1d 参与合并但不剪枝:
对归一化系数和门控参数做幅度剪枝是破坏性的。26 个分片全部通过 ‖out‖/‖base‖ ∈ [0.95, 1.08];逐张量对账 1199/1199,形状与 dtype 零偏差,
全 BF16;config 与 tokenizer 可加载,chat 模板存在;A100 上 13 条 prompt 的 vLLM 冒烟测试。
冒烟测试重点结论:trivial 问题没有被 <think> 刷屏(「你好」27 token、「法国首都」28 token),
可验证题全对(同余 → 23,排名 → B D A E C,三角恒等式走的是三倍角正路),
三条纯中文长文本流畅无乱码无中英夹杂(这是本次合并动了 embed_tokens 后最该查的风险点),
代码题 LRUCache 与 GIL 竞态诊断均正常。
λ₂ = 0)。