ternary-bonsai-27b-gguf

Ternary-Bonsai-27B GGUF (NeuroGrid mirror)

A mirror of the prism-ml/Ternary-Bonsai-27B-gguf weights (Apache-2.0, base Qwen/Qwen3.6-27B), kept here because these are the exact files the NeuroGrid engine serves in production and pinning them makes a deployment reproducible. The weights are not ours and are not modified. Credit and licence belong upstream.

Which file to download

Three quantized files with two different block geometries share the same 2-bit weights. Picking the wrong one gets you a load error, not bad output.

The Q2_0 name means two different byte layouts depending on who wrote the file: upstream settled on group 64, the PrismML fork uses group 128. Both are here so you do not have to guess. NeuroGrid reads both and refuses a file its device kernel cannot run at load time rather than serving it wrong.

• Serving on NeuroGrid: take Q2_0 (group 128), which is what the device GEMV kernel wants.
• Comparing against llama.cpp: take Q2_g64.

Running it on NeuroGrid

One process serves chat, vision and embeddings:

neurogrid -solo \
  -bonsai-gguf   Ternary-Bonsai-27B-Q2_0.gguf \
  -bonsai-mmproj Ternary-Bonsai-27B-mmproj-Q8_0.gguf \
  -bonsai-max-seq 16384 -max-seq-len 16384 \
  -http-port 8080 -host 127.0.0.1

Vision is enabled by the tower being loaded, not by the model name you advertise, so -bonsai-mmproj is the whole switch. Images arrive as OpenAI-style image_url content parts on /v1/chat/completions.

On CUDA the useful environment is NG_QWEN35_GPU=1, NG_QWEN35_GPU_LMHEAD=1, NG_QWEN35_GRAPH=1 and NG_CLIP_GPU=1; add NG_QWEN35_BATCH=8 with NG_QWEN35_BATCH_GRAPH=1 to run up to 8 concurrent sequences through one decode step.

Running it on llama.cpp

llama-server -m Ternary-Bonsai-27B-Q2_g64.gguf \
  --mmproj Ternary-Bonsai-27B-mmproj-Q8_0.gguf -c 16384

Group-64 only. The 7.17 GB files will not load on ggml-org/llama.cpp master.