Qwen3.8-Whittle-dev

Whittle dev: how a 27B was taken apart and what it cost

☕ Support this work

Whittle is built by one person on a grocery budget and rented GPU hours. If this research is useful to you, or you want to see it finished: ko-fi.com/davida81328. Every hour of GPU time goes straight into the next checkpoint, and every checkpoint, table and log lands in these repos.

The developer side of the Whittle family. Every shape we built, what it scored, and the measurements that decided each step. If you just want a model to run, use Qwen3.8-Whittle-tri-14.7B.

Status

Earlier line (Aug 2026); the current Whittle models are the Whittle-Next / Whittle-Qwen-3.8 line: Whittle-Qwen-3.8-35B-A3B.

Needs post training. A research preview, published as the record of a compression method and its measurements rather than as a finished assistant. Further healing and instruction tuning runs are planned over time, and these checkpoints will improve as those land.

What is in this repo

• Scripts at the root: a100_sft_tri.py, battery_kd_colab.py, chat_kd_colab.py, eval_kd_colab.py, kd_student_train.py, kd_teacher_pass.py, q38_battery2.py; chat_template.jinja, chat_template_nothink.jinja, generation_config.json.
• moe/: moe27_build.py, moe27_plan.py, moe_cluster.py, kd_moe_train_v2.py, kd_moe_train_v3.py, kd_moe_train_v4.py, antiloop_mix.jsonl, moe27_plan.pt, moe27_plan_gen2.pt, q38_battery2.py.
• plans/: moe27_plan.pt, moe27_plan_s1024.pt, moe27_plan_s3072.pt, moe27_plan_s5120.pt, moe_mask_sidecar.pt.
• moe-heal-lora/ (adapter, masks.pt, its own README) and tri-heal-lora/ (adapter).
• data/heal60_mix.jsonl.

Recipes, adapters and mixes live in the model repo (Qwen3.8-Whittle-tri-14.7B) under recipe/, heal-lora/, repair-lora/ and training/. The relay, assembler, distillation proof of concept, norm patcher and trainers are all there and all runnable on two 12GB cards.

Measured: the ladder

Fifteen shapes, all measured against the same 80-probe knowledge atlas and the same 39-prompt generation battery.

Findings that changed what we built

Knowledge is not where the general lens says it is. A four-prompt lens priced blocks 8-11 as nearly free. Domain-targeted probes showed those blocks carry web-dev and rare-identifier retrieval: restoring them alone scored above the intact model. Price per domain before dropping anything.

Stopping is a measurable organ. P(end-of-turn) at a finished answer falls from 0.53 in the intact model to 0.05 in the worst cuts. That number is looping, seen from the inside. Restoring blocks 8-11 took it back to 0.55.

Attention alone computes nothing. Keep the 16 attention stations, delete every recurrent layer, and the output is flat noise. The recurrent tissue does the work; attention routes it.

Placement beats amount. Three cuts at identical size and identical selection rule score 0.648, 0.544 and 0.379 on the atlas. Only the attention spacing differs.

Absorbing beats deleting. Distilling a full-attention station into its two recurrent neighbours costs 0.0045 relative error; deleting it costs 0.035.

Sequential distillation compounds, and a global objective fixes it. Per-stage error rose 70x across 24 chained stages and the assembled model scored 12/39. Ninety minutes of global fine-tuning took it to 31/39. The knowledge was mis-coordinated, not destroyed: forced-choice recognition went 0/7 to 5/7.

Width damage is dormant, depth damage is erased. Halve every MLP and the model still recognises answers it can no longer recall (6/7, 7/7). Cut layers and the knowledge is simply gone.

Traps worth inheriting

Synthetic drills that state a rule in prose teach the phrasing, not the skill: our repaired model recites "multiplication binds before addition" and still answers 30 for 10 + 2 * 5. Replay slices amplify whatever format they contain. peft writes a local base path into adapter metadata that HF rejects. modules_to_save norms are trained but carried by neither the LoRA path nor a naive merge. llama.cpp cannot apply LoRA to this architecture at all: the fused GDN projection reshape has no low-rank equivalent, so merging is mandatory.

Collections

Models you can run has the two variants with published, tested weights. Research and dev has everything else, including variants whose weights are not uploaded yet.

Support this work

☕ ko-fi.com/davida81328

Provenance, licence and authors

Base model by the Qwen team (Apache 2.0). Measured by David Aylward with Claude (Fable 5, Anthropic) as co-author.