whittle-next-moe-test

<img src="whittle.svg" width="200" align="right"/>

Whittle-Next MoE (test) — 14.7B dense → 240 experts, ~33% active

☕ Support this work

Whittle is built by one person on a grocery budget and rented GPU hours. If this research is useful to you, or you want to see it finished: ko-fi.com/davida81328. Every hour of GPU time goes straight into the next checkpoint, and every checkpoint, table and log lands in these repos.

A training-free MoE carve of Qwen3.8-Whittle-tri-14.7B, followed by a short learned-assignment alignment pass. No expert weight was ever retrained — only which neurons sit in which expert, and the router.

Ships as Qwen3_5MoeForCausalLM, the stock architecture, so it loads in plain transformers with no custom code and no patches.

Status

Experimental test model — a research artifact, not an assistant; superseded by Whittle-Next-27B-A3B (the parent of the current Whittle-Qwen-3.8-35B-A3B). It is published as the record of an experiment: turning a dense model into a Mixture-of-Experts without retraining the weights, then learning the neuron→expert assignment. It has had no instruction tuning, no chat healing, and no safety work as a MoE. Expect rough edges. Do not use it in production. Use it to reproduce or extend the method.

Which weights are which

• Root safetensors (7 shards) and whittle-next-moe-test-Q4_K_M.gguf: the reference carve at k=58, the one the method section describes.
• kd-k104/ (7 shards) and whittle-next-moe-kd-k104-Q4_K_M.gguf: a better variant that ships alongside the reference carve (added 30 Aug). It runs the same weights at k=104 with a short top-64 KD pass, and cuts the loop-test failure rate from 92.7% to 83.7% (246 paired generations, exact McNemar p = 0.0032). It is still a failing conversational model — just measurably less so. See Follow-up (30 Aug) under Measured for the full A/B.
• Also here: ab_significance.json, loop_test_results.json, whittle.svg.

Measured

Held-out CE and 5-probe factual battery

Same evaluation throughout; "baseline" is the identical carve before alignment (see How it was made).

Alignment is worth roughly 0.8 CE at every sparsity, and the aligned model at k=58 beats the un-aligned one at k=104 — better quality from half the active experts. At full k both reproduce the dense model exactly, which is the carve's correctness check.

Loop test (measured, not estimated)

Run with the standard Whittle loop_test.py harness against this repo's Q4_K_M on an A100, ~90 generations. These are FAILURE rates — lower is better.

For context, the instruction-tuned Whittle v2.1 release passed at 8 / 22 / 7 / 8%.

Read this plainly: as a conversational model, this checkpoint fails. It loops and it answers too briefly. That is the expected consequence of what it is — a training-free carve plus a short alignment pass, with no instruction tuning and no chat healing as an MoE. Its strengths are narrow and real (held-out CE 3.91 at 33% active experts, 5/5 on the factual smoke battery); its conversational behaviour is not one of them. Use it to reproduce or extend the carve method, not as an assistant.

Follow-up (30 Aug): what actually fixes the loop test — capacity, not weights

The loop-test numbers above are bad, so we tried to fix them. Five arms were measured on the same harness, same host, same Q4_K_M quantisation, in one session. The baseline was re-measured rather than quoted, and it reproduced the published table exactly (31/36, 18/18, 26/28, 12/12), so these comparisons are like-for-like.

Failures out of 94 generations — lower is better.

The result, with a significance test. Raising the number of active experts — a pure config change, no training — beat every training run we did. Going from k=58 to k=104 removed 7 failures; the KD pass on top removed 6 more. Because 94 generations is not much evidence for a ~10% shift, the best arm was re-run against the baseline over three independent seed blocks (246 paired generations) and tested with an exact paired McNemar:

So the effect is real. On the single 94-generation run it was not significant (p = 0.12) — worth knowing if you use this harness to gate anything. Late-turn failures fell from 12/12 to 7/12. On an 8-prompt greedy probe the same effect is visible directly: 4-gram repetition halves, 0.227 at k=58 → 0.106 at k=104 → 0.058 at full k. For this carve, conversational degeneration is a capacity problem: at 33% active FFN the model does not have enough width to stay coherent, and it loops.

The cost is compute, not parameters: k=104 activates 2048 + 104×64 = 8,704 / 17,408 ≈ 50% of the FFN instead of 33%. The weights are identical; only down_proj is rescaled by 104/58 to match the runtime's 1/k renormalisation.

Two traps worth knowing if you fine-tune this. Never train mlp.gate.weight. The serving semantics are baked in by scaling every router by 1e-4 (that is what makes the top-k softmax uniform). Those weights are ~1e-4 of normal magnitude, so any normal optimizer step re-peaks the router and down_proj × k/16 then amplifies the damage. Same data, only the trainable set differing: routers trained → CE 4.41, battery 1/5, rep4 0.843; routers frozen → CE 3.946, battery 4/5. Both Muon and AdamW failed identically, so the optimizer was a red herring — the trainable set was the bug.

A short greedy probe is not a chat metric. During chat SFT, held-out CE fell monotonically (3.9522 → 3.9321) while 4-gram repetition on a 3-prompt probe swung 0.446 → 0.014 → 0.831. The checkpoint that looked repaired on the probe scored exactly the same 87/94 as the untrained baseline on the full harness. Gate on the harness, not the probe.

Where the remaining ceiling is. Because the carve is exact at full k, the checkpoint contains its own dense teacher: set top_k = 240 and scale the routed output by 240/58 (never the shared expert). Doing that recovers the dense parent — and the dense parent also fails to stop on 3 of 8 prompts, answers "write a short paragraph about why the sky appears blue" with a JSON write_file tool call, and reads "three tips for writing clearer code" as a three.js request. The carve is faithful to a parent that was never instruction-tuned. The dominant surviving failure is short on multi-turn, and those answers are not truncated — they are confidently wrong and tiny ("In macOS macOS."). That is inherited, and no amount of routing or self-distillation removes it; it needs a better parent or real instruction tuning.

Scripts for every arm (rebake_k.py, teacher_gate.py, train_kd_sharpen.py, train_chat.py, publish_prep.py, loop_test.py) and all raw logs are in logic65/mini-next-a100-kit (private at the time of writing).

How it was made

1. Activation calibration. Token-averaged post-SiLU gate activations were recorded for all 17,408 FFN neurons per layer over a 240-sample, 10-cluster, 40-task calibration set (task diversity, following ExpertWeaver's finding that it beats generic corpus text).
2. Carve. Per layer, the most universal neurons (low coefficient of variation across tasks, high mean activation) became a 2048-wide always-on shared expert; the remaining 15,360 were grouped by balanced k-means on their activation profiles into 240 routed experts of 64 neurons. The router was built without training, as the mean of each cluster's gate vectors. A neuron's gate/up/down slices always move together, so the carve is a pure permutation — exact at full k.
3. Alignment. The neuron→expert assignment was then learned as a balanced optimal transport problem (log-domain Sinkhorn + straight-through estimator), trained against the original dense model as its own teacher. Dense weights stayed frozen throughout.

Geometry

Run it

from transformers import AutoModelForCausalLM, AutoTokenizer

# the reference carve, k=58 (what the method section describes)
m = AutoModelForCausalLM.from_pretrained("logic65/whittle-next-moe-test",
        dtype="bfloat16", device_map="auto")

# the better-measured variant, k=104 + top-64 KD  (92.7% -> 83.7% loop-test failures)
m = AutoModelForCausalLM.from_pretrained("logic65/whittle-next-moe-test",
        subfolder="kd-k104", dtype="bfloat16", device_map="auto")

GGUF: whittle-next-moe-test-Q4_K_M.gguf (k=58 reference) and whittle-next-moe-kd-k104-Q4_K_M.gguf (k=104, recommended). Both run on stock llama.cpp.

To try another sparsity yourself, use rebake_k.py from the kit: it rescales down_proj by k_new/k_old and rewrites num_experts_per_tok. Nothing else needs to change — the router is already flat, and top-k selection is scale-invariant. A CUDA GPU is required — the gated-delta-net layers use Triton kernels.

Caveats

• Test model. No instruction tuning as a MoE. Chat quality is not characterised.
• Held-out CE at the shipped k is still ~0.43 above the dense parent (3.91 vs 3.485). The sparsity is not free.
• The 5-probe battery is a smoke test, not a benchmark. No MMLU/GSM8K numbers are claimed.
• Alignment ran for a few thousand steps at batch size 1 — far below the reference recipe. More alignment is the obvious next lever.
• The serving configuration is baked into the weights (uniform top-k weighting via a scaled router, with down_proj compensating). Change num_experts_per_tok and you must rescale down_proj by k_new / 58, or the output amplitude will be wrong.

Provenance and method credits

Carve follows ExpertWeaver (arXiv 2602.15521); learned assignment follows DOT-MoE (arXiv 2606.01666). Parent lineage: Qwen3.8-27B (Apache-2.0) → Whittle-tri-14.7B → this.

Built by David Aylward with Claude (Anthropic) as co-author.

If this is useful to you: ko-fi.com/davida81328