Bonsai-shonin-27b

Bonsai-27B — 52 blocks, vocab-trimmed, distilled (Q1_0)

2,936,614,240 bytes = 2.937 GB (2.735 GiB). A 27B parameter model under 3 GB, in stock Q1_0 — loads in LM Studio and recent llama.cpp with no patching.

⚠ Read this before using it

pass@1 = 7/150 = 4.7%, versus 9.3% for the 56-block variant at 3.151 GB. This model buys its 214 MB by going one depth group below the measured capability cliff, and it writes noticeably worse code as a result. If 3.15 GB fits your budget, use Bonsai-Shohin-27B-Q1_0 instead. This release exists for the case where under 3 GB is a hard requirement.

Measured capability

pass@1 by execution on newfacade/LeetCodeDataset — each generated solution runs against the problem's own asserts. Greedy, enable_thinking=False, n=150:

The capability cliff sits between 56 and 52 blocks — that gap is real and significant (z=2.75, p=0.006). Distillation moved this depth 2.0% → 4.7%, but at n=150 that is not significant (z=-1.29, p=0.20); treat it as suggestive, not established.

How it was built

1. Depth prune 64 → 52 blocks, dropping groups 5, 7 and 9 (original blocks 20–23, 28–31, 36–39), chosen by leave-one-out perplexity then greedy search. Layer type derives from index mod 4 (qwen35.full_attention_interval = 4), so only whole 4-block groups can be removed without breaking Bonsai's 3:1 linear/full attention phase — which is why 53/54/55 are not freely constructible and 52 is the next stop below 56.

2. Logit distillation from Qwen3.5-35B-A3B. Bonsai and every Qwen3.5 model share an identical 248,077-token vocabulary — same strings, same ids, same tokenisation — so this is true KL distillation against the teacher's output distribution, not SFT on teacher text. Cached top-64 log-probs over 2.0 M tokens of resolved SWE-smith agent trajectories; loss is KL over the teacher's top-64 support (renormalised) plus 0.1× cross-entropy on the true next token. Only the 186 M group scales were trained, as scale₀·exp(δ), so zero bytes were added. 300 steps, lr 3e-3. Trajectory perplexity 5.021 → 3.409 (−32%).

3. Vocabulary 248,320 → 98,782. Over 11.9 M tokens of real traffic only 66,741 distinct tokens (26.9%) were ever used. Kept those plus low-numbered unused ids as headroom, with a transitive closure over the merge table (+478 ids) so every kept token stays reachable by BPE — without it, 518 rows would exist but be unemittable, silently degrading tokenisation. Embedding rows copied bit-exact.

Usage

lms load <model> --gpu max --context-length 8192 --parallel 1

Bonsai is a reasoning model. Budget ≥2500 output tokens, ideally 6000, or the reply is consumed by chain-of-thought and content returns empty (75% of replies were trapped at a 1200-token budget; 0% at 2500).

KV cache dominates VRAM at this weight size — ~54 KiB/token across the 13 full-attention blocks. --parallel 4 quadruples it. Runs on an 8 GB card.

The useful negative result

Perplexity does not track coding capability on this model family. Four attempts:

At 48 blocks, healing beat the 3.803 GB parent on trajectory perplexity (3.06 vs 4.51) while solving zero problems the parent solved seven of — syntactically valid Python rose 70% → 80% with pass@1 pinned at 0. It learned to write more plausible wrong code.

Mechanism: depth pruning removes computation; training group scales adjusts calibration. Scales are 0.8% of the model and can only rescale groups of 128 weights — never change which direction a weight points. Valid-Python rate is not a proxy either: it held at 87.5% even at 2.5% pass@1.

Caveats

• 4.7% pass@1 is low. This is a size-constrained release, not a capable coding model.
• Not run through llama.cpp's own binary; validated in LM Studio and a PyTorch loader.
• Trimmed vocab drops CJK, Cyrillic and emoji — the measured corpus was English + code.
• A trimmed model cannot use the original 248,320-token tokenizer; that indexes past the embedding and aborts with a CUDA device-side assert. Use the embedded one.
• Long context untested — all perplexity used 2048-token windows, and pruning removed 3 of 16 full-attention blocks.
• n=150 on one benchmark; no SWE-bench, no agentic or multi-turn evaluation.
• Text-only; no mmproj vision projector.

License

Apache 2.0, inherited from prism-ml/Bonsai-27B-gguf. Base model by Prism ML, derived from Qwen3.6-27B; teacher logits from Qwen3.5-35B-A3B. This repository contributes only depth pruning, vocabulary trimming, scale distillation, and measurement.