Qwen3.8-Flash-Next-REAM-288

Qwen3.8-Flash-Next-REAM-288

Qwen3.8-Flash-Next with 288 of 512 experts per MoE layer, merged not pruned (REAM), in native bf16 for transformers. 251GB vs 360GB stock. MTP block included (its experts merged 512->288 as well) — vLLM-style qwen4_exp speculative decoding keeps working.

Method

REAM (arXiv 2604.04356, Samsung SAIL Montréal: Merging Improves Pruning of Experts in LLMs), offline adaptation: instead of dropping the 224 non-kept experts per layer (REAP), each is merged into its most-similar kept expert with saliency-proportional coefficients.

• Kept set / slot order: exactly the per-layer REAP-288 manifest published by sh0wie (saliency calibrated on ~686K tokens of agentic-coding traffic) — full credit to that work; this build is a direct A/B against it.
• Similarity (offline, no calibration run): cosine of router gate rows.
• Merge coefficients: measured saliency from sh0wie's saliency_full_505.npz (its argsort reproduces his pinned manifest bit-for-bit), tempered and shrunk: w_i = (S_i/S_c)^2 * router_cos * count_confidence, with the centroid keeping at least 90% of itself. Raw proportional weights are catastrophic here (KL 1.397) because the real kept/dropped saliency gap is only ~1.6x — the merge dissolves the expert REAP chose to keep. Hungarian permutation alignment of intermediate channels before averaging (fp32 accumulate, bf16 out).
• Trajectory flat_tsg_tsD: flat one-shot groups (canonical REAM)
• Router: absorbed rows deleted; kept rows byte-identical to stock.
• MTP block: same machinery, k-center centroids on its own gate rows.
• A six-point ladder (see campaign/reports/truesal_ladder.md) locates the optimum: KL is minimised when kept experts give away ~9% of their mass — both heavier blending and near-pruning are worse.

Evaluation (A/B vs REAP prune, same harness, same hardware)

KL: teacher-forced top-256 KL vs the stock model over a fixed mixed corpus (en/ru/code/rare-entities/agentic, ~10K positions). Probe: rare entities x 10 seeded sampled generations, pass = name reproduced intact; aggregates recomputed on the name subset common to all rows (a monitoring bug killed the winner's third probe batch mid-run). HumanEval: 25-task slices of a frozen 50-task subset, no-thinking, greedy; the two 288e models landed on different slices (2-task overlap), so each is paired with the stock score on its own slice. Single-run evals; ±1-2 point differences are noise. RU slice of the probe (3 Cyrillic names, 30 gens): stock 27, prune 21, merge 24.

4-bit MLX head-to-head (identical harness)

These rows are the 4-bit builds, not this one. This card carries them because they are the campaign's only like-for-like merge-vs-prune comparison — same script, same GPU, same problems. This build was not benchmarked generatively (the bf16 generative evals need a pod with >=160GB VRAM); judge it by the bf16 selection evidence above and by the 4-bit result as evidence about the merge itself, not about this file.

Both published 4-bit builds were run through the same script (eval_mlx.py / probe_mlx.py on mlx-vlm), on the same GPU, over the same problems with the same prompts, chat template, decoding and checkers — only the checkpoint differs. HumanEval is the full 164; GSM8K is 200 problems drawn evenly over the whole test split.

How to read this:

• Neither difference is significant. On code, paired McNemar gives p = 1.000 (4 problems solved only by the merge, 3 only by the prune, 14 defeat both). On maths, p = 0.585 (17 vs 13). At these sample sizes the merge and the prune are the same model on both benchmarks.
• The earlier 25-problem numbers were noise, and are retracted here. Two non-overlapping 25-problem GSM8K draws put one build at 72% and 40%; the pooled 29/50-vs-24/50 edge (p = 0.18) that an earlier version of this card reported did not survive 200 problems.
• What does survive is KL: 1.2564 for the merge against 1.3066 for the prune, measured against the stock 512-expert model. That is the one metric where the merge is consistently ahead.
• GSM8K numbers are capped by the token budget. At max_new=320 many completions are cut off before the answer line; of six problems both builds failed at that cap, five pass at 768. The cap applies equally to every row, so it biases the absolutes, not the comparison.
• 4-bit hurts non-Latin names badly, and the merge is not to blame. All three Cyrillic names score 0/5 on both builds, the author's untouched one included. The same merge in bf16 scores 93.3% against 62.7% in 4-bit over the same 15 names — a 30-point gap that belongs entirely to the quantizer. (The bf16 selection table above reports 94.0% for this build over the 10-name subset the selection gate used; both numbers are real, they are just different name sets.) For factual recall on rare non-Latin entities, use bf16.

Files

Standard HF sharding. experts-layerNN shards carry the merged expert stacks; backbone tensors are byte-identical to stock (verified by streaming sha256).

Limitations

• Merge coefficients use the author's measured saliency, tempered (see Method).
• Calibration-free similarity (router-row cosine): no activation statistics.
• MTP merged without measured saliency (k-center on gate rows).
• Vision tower untouched but untested after merging.

Credits

• sh0wie / HamsterResearch — REAP-288 saliency calibration, kept-expert manifests, MLX conversion conventions.
• Samsung SAIL Montréal — the REAM method.
• Qwen team — the base model. License inherited (Qwen Community License 1.0).