Binary-Bonsai-27B-int4-sym-ov

Binary Bonsai 27B — OpenVINO INT4 SYM

This is an OpenVINO INT4 conversion of PrismML's Bonsai-27B unpacked checkpoint, packaged for Intel CPU/GPU inference.

PrismML created and trained Bonsai. This repository contains only the OpenVINO conversion by Wondernutts. Please read and credit the original model card for the model's background and intended use.

Important: this is not the native 1-bit build

The source checkpoint expands Bonsai into ordinary FP16 tensors for compatibility with stock Hugging Face tooling. This repository quantizes those unpacked tensors to conventional OpenVINO INT4. It does not preserve PrismML's native 1-bit packing, kernels, or memory advantage.

For the smallest and fastest native Bonsai deployment, use PrismML's 1-bit MLX or 1-bit GGUF releases instead.

Conversion details

The repository includes the language model, text embeddings, vision components, tokenizer, and detokenizer IRs. The advertised architectural context limit comes from the source configuration and has not been independently validated for this conversion.

Download

Download the public repository:

python -m pip install "openvino-genai==2026.2.1.0" "huggingface_hub>=1.0"
hf download Wondernutts/Binary-Bonsai-27B-int4-sym-ov --local-dir ./binary-bonsai-27b-int4-ov

Run on Intel GPU

VLMPipeline is required because this export contains the conditional-generation/VLM graph. Text-only chat works without supplying an image.

import openvino_genai as ov_genai

model_dir = "./binary-bonsai-27b-int4-ov"
pipe = ov_genai.VLMPipeline(model_dir, "GPU", CACHE_DIR="./ov_cache")

config = ov_genai.GenerationConfig()
config.max_new_tokens = 512
config.do_sample = True
config.temperature = 0.8
config.top_p = 0.95

pipe.start_chat("You are a concise, capable assistant.")
print(pipe.generate("Explain why the sky changes color at sunset.", generation_config=config))
pipe.finish_chat()

start_chat() uses the tokenizer/chat template shipped with the model and retains KV cache across turns. Change "GPU" to "CPU" for CPU inference. If you build a continuous-batching server on Intel GPU and encounter garbled output, disable dynamic activation quantization with DYNAMIC_QUANTIZATION_GROUP_SIZE=0 in the device/pipeline properties.

Optional image input

import numpy as np
from PIL import Image
from openvino import Tensor

image = np.asarray(Image.open("image.jpg").convert("RGB"), dtype=np.uint8)[None, ...]
print(pipe.generate("Describe this image.", images=[Tensor(image)], generation_config=config))

Vision capability and long-context behavior have not been independently benchmarked for this conversion.

License and attribution

Apache-2.0, following the source repository. Model authorship and training credit belong to PrismML; OpenVINO conversion credit belongs to Wondernutts.