Qwen3.5-4B-graft16

Qwen3.5-4B-graft16

A graft of Qwen/Qwen3.5-4B: the model is cut at layer 16 of 32, and only the lower half reads your prompt. The upper half never sees it. Instead, the residual stream the lower half produces becomes a memory, and a small per-site adapter maps that memory into the keys and values each upper attention layer would have computed for itself. Generated tokens still pass through all 32 layers, so decoding costs exactly what the original model costs.

Time-to-first-token is 1.8-1.95x faster from 16K tokens up, and decoding is unchanged. 1.81B parameters run per prompt token instead of 4.21B.

A40 48 GB, bf16, flash-attention-2, batch 1, greedy, 7 timed reps after 2 warmups per length; median reported (min and p95 within 1% of it). TTFT is prefill plus the first sampled token. Decode is the steady-state rate over the next 31 tokens and is the same for both models by construction, since generated tokens still cross all 32 layers. bench_ttft.py, included in this repo, reproduces the table.

Two things this does not buy: the KV cache is the same size (the encoder's own attention keys and values are not compressed, so peak memory is within 1-6%), and short prompts gain least.

This is one instance of a general operation we call model grafting — cutting a trained model, changing how information moves through it, and healing the result with a small amount of continued training. The method, the cost law that predicts how much healing a given change needs, and the study this model came out of are written up here:

→ Model grafting

What it is for

Long prompts, where most of the wait is the model reading what you wrote. The graft holds general capability close to its parent and is noticeably weaker at one thing: following chains of references through a long context (see the numbers below). Read it before deploying this in an agent loop that chases references across a long document.

Results

Six benchmarks, greedy decoding, thinking disabled, identical prompts for both models. MMLU 5-shot (50 per subject), GSM8K 5-shot (400 questions), IFEval (541 prompts), HumanEval-instruct, BFCL v4 non-live AST (100 per category, official bfcl_eval checker), HashHop (Magic's generator, 25 instances per hop count at 1–4 hops, ~8K-token prompts).

Where the gaps are:

• Parallel tool calls. BFCL by category: simple 0.89/0.89, multiple 0.91/0.94, parallel 0.70/0.73, parallel-multiple 0.78/0.83, irrelevance 0.89/0.91 (graft/parent). Single calls are at parity; emitting several calls in one reply is where it loses ground.
• Multi-hop retrieval, and it gets worse with length. At ~8K tokens, one hop scores 0.48 against the parent's 0.88. At ~33K the graft answers almost nothing (0.01 against 0.10). The parent is also weak here (neither model solves 3 or 4 hops), but the graft is clearly worse at exactly the prompt lengths where its speed advantage is largest. This is the honest limit of the model.

Speed is measured at the top of this card. The capability cost above is what the 1.8-1.95x prefill speedup buys.

How it was trained

Healed on 500M tokens on a single A40, in two stages (100M, then a re-warmed continuation to 500M):

• Objective: self-distillation only. The target is the unmodified Qwen3.5-4B's own next-token distribution (top-64 KL, kd_lambda 1.0), not the data's labels. That is what protects instruction following, tool-call format and thinking mode, which plain continued pretraining erodes.
• Data mix, per sequence: 45% agent trajectories (UltraData-SFT-Agent-2609, rendered with Qwen3.5's own chat template), 15% code (UltraData-Code L2, quality ≥ 5, HumanEval-decontaminated), 40% FineWeb-Edu. Sequence lengths 4K/8K/16K mixed 20/40/40.
• What trains: the 16 encoder layers, the four memory-site attention modules, and the adapters (1.96B of 4.23B parameters). Everything above the cut is frozen — that both prevents capability erosion and cuts optimizer memory.
• Adapters: one identity-initialised 2560×2560 linear map per memory site (layers 19, 23, 27, 31), 26M parameters total. The adapter does not predict keys and values; it maps memory into the host layer's input space and lets that layer's own input_layernorm, k_proj, k_norm, v_proj and RoPE do the rest.
• Tail window: 256. The last 256 prompt tokens go through the full decoder path rather than through memory. Without this the upper recurrent layers have no state over the question and retrieval fails outright.
• Hyperparameters: AdamW8bit, lr 3e-6 (pretrained) / 1.2e-5 (adapters) cosine to 10%, batch 262144 tokens, grad-checkpointing, bf16 frozen weights with fp32 masters, ~1,840 tok/s, 45 h for the second stage.

One finding worth repeating from the run: the distillation loss on held-out agent data fell monotonically (0.58 → 0.41) over the last 300M tokens while no downstream benchmark improved by more than a point. Going from 100M to 500M tokens moved the six-benchmark mean by 0.7 points. The plateau arrives early; more tokens at this split is not the lever.

Loading it

This checkpoint is not a drop-in transformers model. The loader it needs, modeling_ced.py, is included in this repo, and the weights file stores only the parameters that were trained — the trainable_only marker means the frozen ones are loaded from Qwen/Qwen3.5-4B (recorded in ced_config.json), so the base model has to be available too.

from huggingface_hub import snapshot_download
from transformers import AutoTokenizer
import sys

d = snapshot_download("LocalLLaMA/Qwen3.5-4B-graft16")
sys.path.insert(0, d)                            # modeling_ced.py ships with the weights
from modeling_ced import CEDForCausalLM
model = CEDForCausalLM.load(d)                   # frozen weights come from Qwen/Qwen3.5-4B
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-4B")

prompt = tok.apply_chat_template([{"role": "user", "content": "..."}], tokenize=False,
                                 add_generation_prompt=True, enable_thinking=False)
ids = tok(prompt, return_tensors="pt").input_ids.cuda()
out = model.generate(ids, max_new_tokens=256, eos_id=tok.convert_tokens_to_ids("<|im_end|>"))
print(tok.decode(out[0], skip_special_tokens=True))

generate prefills the prompt through the encoder once, builds the memory, and then decodes through the full stack. model.cfg.tail (256) is the suffix that bypasses memory.

Files

Limits

• One cut depth at one scale. A shallower cut is faster and worse at retrieval; the frontier is in the article.
• Long-context multi-hop retrieval is the failure mode, and it is worst where the graft is most useful.
• Benchmarks use capped sample sizes (50 MMLU questions per subject, 400 GSM8K, 100 BFCL per category, 25 HashHop per hop count), so treat differences under about a point as noise.
• Inherits Qwen3.5-4B's licence and limitations. The vision tower of the parent is untouched and untested here; everything above was measured text-only.