Bonsai-2-27B-q27

Ternary Bonsai 2 27B -- q27 pack

PrismML's Ternary Bonsai 2 27B repacked for the q27 engine. It is Qwen3.8-27B with every projection ternary (one fp16 scale per 128 weights) in a Hadamard-rotated basis, and no MTP block. This pack is an exact re-encoding of the PTQ1_0 GGUF, not a requantization: every trit is stored as-is in q27's 2-bit T2_G128 container, the embedding and the head as int8 (also exact), and the rotation metadata is carried verbatim so the engine rotates activations itself.

Needs q27 v0.12.0 or later (the MTP pack: v0.13.0) (docs/plans/2026-09-18-bonsai2-ternary.md in the repo has the design). The pack has no MTP head, so DFlash2 is the only speculative path. The Qwen3.8 drafter packs in signalnine/Qwen3.8-27B-MTP-q27 work against this target as-is (3.47 tokens per round on Claude Code traffic); a drafter trained on the ternary target does better -- ProCreations' Ternary-Bonsai-2-27B-DFlash2 (independent, Apache-2.0) packs unchanged with python3 tools/dflash2_pack.py <its dir> bonsai2-dflash2-q8-serve.d2w --q8 and gives 3.80 tokens per round, 227.7 t/s aggregate on the same run.

The MTP pack (multi-slot)

bonsai2-27b-t2-mtp.q27 is the pure pack plus ProCreations' Ternary-Bonsai-2-27B-MTP head (independent work, Apache-2.0, not a PrismML release): the Qwen3.8 MTP block distilled onto Bonsai's hidden states, stored here as the pack's blk.64 (Q8 matmuls, F32 norms, kept unrotated -- the engine rotates nothing on that layer). Every other tensor is byte-identical to the pure pack. With it the engine runs its ordinary speculative ladder against the ternary target, which is what makes fused multi-slot rounds pay: on a 5090 at 8 slots / 16K, C = 1 / 2 / 4 / 8 concurrent streams decode at 173 / 231 / 384 / 512 t/s aggregate (the draftless pure pack: 105 / 150 / 200 / 329; the Qwen3.8 q4s tier: 141 / 230 / 352 / 531). Serve it with --slots 8 --ctx 16384 and Q27_BATCH=1, no DFlash2 pack. For one slot the DFlash2 route below is faster (227.7 vs 178.8 t/s on Claude Code traffic), so this pack is the multi-slot choice, not the default. The ladder reproduces plain decode byte for byte over 1500 CLI tokens; load digest wsum: 0f85b21f8721e39c. Rebuild from the two upstream checkpoints: python3 tools/repack.py Ternary-Bonsai-2-27B-PTQ1_0.gguf out.q27 --mtp-safetensors model_mtp.safetensors.

The 8 GB-card packs

Ternary weights carry 1.58 bits each, and the 2-bit container wastes the rest. bonsai2-27b-t3-slim.q27 stores the body as T3_G128: five trits in a byte, base 3, 1.6 bits per weight plus the same fp16 scale per 128 (the layout the q27 Metal backend already read; the CUDA engine reads it since v0.14.0). Same trits, same scales, and a decode kernel built to sum exactly what the 2-bit kernel sums, so the pack is bitwise the T2 pack: every one of the 400 body matrices at GEMV widths 1 through 8, the same wikitext NLL to six digits, the same server texts, the same speculative rounds. The embedding and head stay in T2 (exact, 0.68 GB of the 6.06). bonsai2-27b-t3-mtp-slim.q27 adds the MTP head from the pack above.

Serving on 8 GB. The first row is a real card (an RTX 3060 Ti, headless via init 3, 2026-09-22); the rest are the 3090 with its memory held down:

Claude Code's stock prompt with its tool schemas is larger than a 24K window, so on the 24K and 12K rows run it as claude --bare --system-prompt "."; a headless 8 GB card takes the defaults.

Build make build/q27-server-12g (one sm_86 image, 8 lanes, 256-row prefill), run it with Q27_FIXED_STACK_GB=0.6 (0.8 for the MTP pack), Q27_BATCH=0 and one slot (tools/install-bonsai2-8gb.sh in the repo does all of this on a fresh box); the KV format defaults to turbo5k on Ampere, which is where those context numbers come from. Decode runs at the T2 pack's speed rather than below it -- the digit extraction costs about what the saved bytes bought on a 3090 -- and the 3060 Ti measures 42 t/s plain, 481 tok/s prefill. Load digests: wsum: 3b876779744bd102 (plain), wsum: b9ab2d8221224dfd (MTP). Rebuild: python3 tools/repack.py Ternary-Bonsai-2-27B-PTQ1_0.gguf out.q27 --bonsai2-container t3 --slim (add --mtp-safetensors model_mtp.safetensors for the MTP pack).

How faithful the port is

Bit-exact containers make this checkable against the reference fork on the same GGUF:

• teacher-forced logits vs the fork over 383 positions: top-1 agreement 0.9974, NLL 2.0792 vs 2.0796
• wikitext-2 perplexity at 2048: q27 8.2767 vs llama-perplexity 8.2643 (q27 runs fp8 KV and int8 activations, which is the 0.15%)
• the 2-bit prefill GEMM is bitwise the int8 tensor-core GEMM on an exact Q4 image of the same matrices, and the full-corpus NLL is identical to six digits through both paths

What the checkpoint measures like

Same instruments as the Qwen3.8 tier table, so read these side by side:

On a 12-instance Claude Code SWE-bench run it lands the same patches as the Qwen3.8 tiers (gold 11/12, identical miss) at 205 vs 222 t/s aggregate, but it reasons about twice as long per instance -- 37 vs 22 API turns, 75K vs 35K thinking characters -- so wall time is 2.3x. It is a different checkpoint, not a smaller tier of the model above; pick it for the 9 GB footprint and the context that buys, not for quality.

Serving

hf download signalnine/Bonsai-2-27B-q27 --local-dir models/bonsai2
hf download signalnine/Qwen3.8-27B-MTP-q27 qwen38-dflash2-q8-serve.d2w --local-dir models/qwen38
(cd models/bonsai2 && md5sum -c CHECKSUMS.md5)

Q27_KV=fp8 Q27_BATCH=0 Q27_DFLASH2=models/qwen38/qwen38-dflash2-q8-serve.d2w Q27_DFLASH2_RESERVE_GB=3 \
  ./build/q27-server models/bonsai2/bonsai2-27b-t2.q27 models/bonsai2/qwen38-27b-mtp.tok \
  --think --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.05 --think-budget 0

That is the Qwen3.8 agentic recipe; the engine detects the pack, turns the activation rotation on and the MTP ladder off, and selects the 3.8 XML tool dialect from the name. Weight digest on load (Q27_PRINT_WSUM=1): wsum: 9161ec16da4b66a1. To rebuild the pack from the upstream GGUF: python3 tools/repack.py Ternary-Bonsai-2-27B-PTQ1_0.gguf bonsai2-27b-t2.q27 (about three minutes, numpy only; the converter verifies every ternary slot).

Upstream checkpoint and license: Apache-2.0, PrismML (prism-ml/Ternary-Bonsai-2-27B-gguf, Ternary-Bonsai-2-27B-PTQ1_0.gguf). The MTP head in bonsai2-27b-t2-mtp.q27 and bonsai2-27b-t3-mtp-slim.q27 is ProCreations' work (ProCreations/Ternary-Bonsai-2-27B-MTP, model_mtp.safetensors, Apache-2.0).