Ternary-Bonsai-2-27B-vllm

Ternary-Bonsai-2-27B-vllm

Work in progress. Expect rough edges and breaking changes.

Bonsai 2 27B by Prism ML, repacked for vLLM. Unofficial; not affiliated with Prism ML.

It needs the prism_ternary vLLM plugin from fraserprice/bonsai-vllm, which has the run command, kernels and throughput numbers:

docker run --rm --gpus all --ipc=host -p 8000:8000 -v bonsai:/cache fraserpricee/bonsai-vllm:20260918

Built and tested for the RTX PRO 6000 Blackwell only; other NVIDIA GPUs are untested. Problems: open an issue.

What's in here

• The ternary weights of the MLX pack, bit for bit: the same 2-bit codes and FP16 group scales (g128), in the same Hadamard-rotated basis.
• Two tensors the MLX pack stores per layer are left out, as they carry no information: biases, which is always exactly -scales (MLX dequantizes as code * scale + bias; ternary is (code - 1) * scale), and signs, the ±1 vector of the Hadamard rotation, which is the same for every layer of a given input width and so is kept once per width in config.json. The converter checks both.
• The embedding table is dequantized to BF16.
• Norms and the linear-attention state path are Prism ML's own tensors from the MLX pack, converted to the Hugging Face layout.
• The MTP head, which the MLX pack doesn't carry, is the BF16 head of Qwen/Qwen3.8-27B. It only drafts tokens for speculative decoding; every token is verified by the Bonsai model.
• Text only: the vision tower is not included.

Produced by prism-ternary-convert from the repo above. For benchmarks, sampling parameters and everything else about the model, see the original model card.

License

Apache 2.0, as the original. Created using Bonsai by Prism ML; built from Qwen3.8-27B by Alibaba Cloud. See NOTICE.txt.