Ornith-1.5-27B-A3B-Coder

Ornith-1.5-27B-A3B-Coder

BF16 weights for Coder — a code-targeted expert prune of Ornith-1.5-35B-A3B: 256 experts per layer reduced to 184, ~35.9B → ~26.7B, still A3B active. Router width, attention, norms and the MTP speculative head are untouched.

This is the pure competence-map arm (p0): selection is driven entirely by our per-category contribution ranking, with no stability floor on top. Its sibling Ornith-1.5-27B-A3B-CoderX keeps the same 184-expert budget and the same map, but rescues 6 experts per layer that REAP ranks highly — and measures better on almost every code and reasoning axis. If you are choosing between the two, start with CoderX.

Recipe

Contribution-based expert selection from a per-category competence map built on a targeted code corpus — not generic activation magnitude — then a pure slice. Nothing is folded or merged: every surviving expert is bit-identical to its counterpart in the base (verify_arm_identity.py → ARM_IDENTITY_OK). No fine-tuning, no distillation.

Maps and recipe (reproducible, in the open)

Everything that produced this checkpoint lives in omnimergekit, under recipes/ornith_1_5_35b_a3b_prune/:

Quantised builds

GGUF (imatrix at every tier, MTP head included): Ornith-1.5-27B-A3B-Coder-MTP-GGUF.

Evaluation

Q6_K + imatrix, llama.cpp, sampler=recommended (temp 0.6 / top_p 0.95 / top_k 20 — the vendor profile for this family; greedy is not viable here), identical templates and identical prompts across the three columns.

Read this as "the cut is free on code, and costs science". Against its own teacher this arm gains +6.5 pp on LiveCodeBench v6, +5.5 pp on LCB-medium and +1.8/+2.4 pp on HumanEval/HumanEval+, while giving up −6.1 pp on GPQA and −2.3 pp on MultiPL-E. Dropping 28% of the experts did not cost code quality; it did cost general-science headroom, and the CoderX sibling recovers most of that (0.8131) for free.

At n=77 one LCB problem is 1.30 pp and the unpaired binomial SE is ≈7.4 pp; at n=100 one problem is 1.0 pp. Treat sub-2-problem gaps as ties.

This arm holds the cohort's only repeat. Two complete, independent 77-problem runs of the LCB cell under the same configuration both scored 0.7273 (56/77) — with visibly different generation profiles (tok_p50 12,680 vs 12,654, 5 vs 4 cap-hits, max response 109,593 vs 125,587 chars). Identical score, different draw. That is one repeat on one cell, not a variance band for the whole table.

Per language, MultiPL-E (Coder / CoderX / 256e): Rust 0.74 / 0.71 / 0.75 · Java 0.87 / 0.92 / 0.91 · JS 0.88 / 0.88 / 0.90.

Mixed measurement basis, deliberately. The LCB cells for Coder and CoderX were measured on a patched llama.cpp in which a PEG parse failure degrades to content instead of throwing (which used to discard the generation entirely); the base cell predates that patch but recorded zero discarded rows, so it is unaffected. Nothing else differs — same quant, same server flags, same sampler, same prompts.

Tool-calling benchmark — tool-eval-bench hardmode (88 scenarios, 176 pts)

Ornith-1.5-27B-A3B-Coder scores 136.8 ±4.8 of 176, eighth of ten, statistically indistinguishable from Qwen3.6-27B-A3B-CoderX (137.4) — the CIs overlap almost completely — and from its own CoderX sibling (134.0).

Note the quant: this row is IQ4_XS, while the A3B rows are Q4_K_M. Part of any gap against them is quantisation, not architecture.

Category profile: perfect (100%) on Tool Selection, Restraint & Refusal, Structured Reasoning, Toolset Scale and Creative Composition, with Structured Output 11.6/12 (96.7%). The weaknesses are the agentic ones — Autonomous Planning 2.8/6 (46.7%), Context & State 12.4/20 (62.0%), Hard Mode 24.6/38 (64.7%) — plus an unusual Parameter Precision 4.0/6 (66.7%), which is markedly worse than every other model in the cohort and is worth knowing before wiring it to strict-schema tools.

12 safety-critical failures across five seeds: TC-60 (Cross-Turn Sleeper Injection) on all five, TC-58 (Fake System Message in File) on four, TC-33 on two. It does not share the A3B models' TC-31/TC-34 weakness.

One seed hits the TC-62 scorer crash described below.

Full cohort

* one seed (s42) is graded on 174 pts, not 176 — see that model's card.

<details> <summary><b>Basis — read before comparing these numbers to anything</b></summary>

• Scorer: tool-eval-bench v2.6.0 (the pip/uv-installed package, verified via tool_eval_bench.__file__, not a git checkout). An earlier note in the runner claimed cf54b4b (v2.6.0-45); that is wrong and has been corrected — no cell ever ran it. All 50 cells ran the same v2.6.0, so the cohort is internally consistent.
• v2.6.0 carries a known scorer crash on TC-62. email_calls[-1] raises IndexError when a model sent no valid CFO email; the orchestrator catches it and returns FAIL / 0 points while keeping the scenario in the denominator. It hits 11 of 38 scored cells, 2 pts each, and it is not neutral — it concentrates on the weakest models. Later harness commits credit that behaviour instead, so a fixed scorer would raise affected scores, unevenly.
• 5 paired seeds [42–46], 64k context, context-pressure 0.25, max 8 turns, 120 s timeout, thinking enabled, sampler temp 0.6 / top-p 0.95 / top-k 20 (not greedy).
• Served on llama.cpp b1788384120-c588c4f47 with MTP speculative decoding enabled (nextn=YES spec=mtp), one model per GPU, sequential.
• Quant tiers are not uniform across the cohort (Q4_K_M for the Omnimerge/A3B rows, IQ4_XS for Ornith and 35B-A3B, UD-Q4_K_M for the Qwen3.8 base). Cross-row gaps therefore carry a quantisation component and are not purely architectural.
• Do not pool these with the r/LocalLLaMA published tool-eval-bench figures: those were run at 256k context and are a different basis despite the shared scorer version.

</details>

Cross-family reference — Qwen3.6-27B-A3B

The sibling program applied the same idea (256e → 184e, competence map, REAP stability floor) to Qwen3.6-35B-A3B. Those cells are sampler- and template-matched to the table above — every one records sampler=recommended, temp 0.6 / top_p 0.95 / top_k 20, Q6_K, llama.cpp — so they can share a table. Read from each run's summary.json, not from the sibling cards.

Only the five benches below ran the same template in both programs. The Qwen cohort used family-specific variants elsewhere (aime_30_qwen, gsm8k_100_boxed, math500_100_qwen, humaneval_full_think), and its cells on the plain templates this program used are not usable (Qwen CoderX scores 0.22 on math500_100 and the base 0.36 on gsm8k_100 — a template/parser mismatch, not capability). Those rows are omitted rather than cross-quoted.

What it says. The floor arm wins in both families — Qwen CoderX beats Qwen Coder by 14.3 pp on LCB, Ornith CoderX beats Ornith Coder by 3.9 pp — which is the strongest evidence that the stability floor is doing real work and not fitting one model's noise. Between families: Qwen CoderX is the better LiveCodeBench model (0.8312 vs 0.7662), Ornith CoderX the better MultiPL-E one (0.8367 vs 0.8000), and Ornith gives up a lot of instruction-following (IFEval 0.7600 vs 0.9300). These are different base models, so the column difference is not attributable to the prune.

Why these differ from the numbers on the Qwen cards. The Qwen program measured two cohorts. Its own cards quote the greedy one (lcb_v6_77q_48k, 24k think / 48k total, template_default T=0.0) — e.g. LCB CoderX 0.727. The table above uses the sampled qwen_suite cohort (lcb_v6_77q, recommended T=0.6), because that is the cohort whose template and sampler match this program's cells. Both cohorts are correct and each is labelled on its own card; they are simply not interchangeable.

Generation length and deliberation

Where this family actually thinks. Nine of the eleven benches run in split reasoning regime and the model answers essentially directly: thinking_tokens_est is 0 on GPQA, AIME, MATH-500, GSM8K, IFEval, HumanEval and MultiPL-E, and rounds to ~1.2% of completion on ARC. Only LiveCodeBench v6 runs inline, with a 12,288-token thinking budget inside a 32,768-token cap — and there the median generation is ~12.7k tokens, an order of magnitude above every other bench. This model ruminates only where the budget invites it.

Completion tokens, p50 / p90 / max, and cap-hits at the per-bench ceiling:

The one real regression is the LCB tail. Median length is indistinguishable across all three arms (12,690 / 12,680 / 12,697 tokens — within 0.1%), but both prunes run away on 5 of 77 problems, hitting the 32,768 ceiling where the base never exceeds 14,296 tokens. Those truncations score as failures, so this arm's 56/77 is achieved despite them. Four of them (lcb/leetcode/{3402,3562,3648,3801}) show truncation-shaped errors — IndentationError: unexpected inden…, NameError: 'total_with' is no… — cut off mid-code rather than wrong. The base solves only 3402 of the four, and fails the other three with complete answers, so this is a length problem on top of a capability gap, not instead of one.

Deliberation vs. answer, on LCB. There is no separable reasoning channel to report: these caches carry no thinking table, and llama.cpp's --reasoning-format deepseek strips the <think> delimiters — only 2–3 of 77 rows retain one. The split below therefore uses LCB's structural anchor: the submitted answer is the last fenced code block, everything before it is deliberation. It is a proxy for the reasoning/answer split, not a token count:

~94–95% of every LiveCodeBench generation is deliberation, and the code that actually gets graded is ~1.4k characters. All three arms deliberate at the same median; pruning did not make this model think more or less, it only widened the tail. If you want that tail bounded, cap it at serve time (--reasoning-budget) rather than hoping for a shorter draw.

Limitations

• Single-draw evals except the one LCB repeat noted above. Do not rank arms on a sub-2-problem margin.
• −6.1 pp GPQA against the teacher is the real cost of this arm; CoderX recovers most of it at the same expert budget.
• The 5-of-77 LCB runaway is a real tail-latency regression, inherited by both prunes.
• Apache-2.0 · research checkpoint.