Ternary-Bonsai-27B-int4-sym-ov

Ternary Bonsai 27B — OpenVINO INT4 SYM

This is an OpenVINO INT4 conversion of PrismML's Ternary-Bonsai-27B unpacked checkpoint, packaged for Intel CPU/GPU inference.

PrismML created and trained Ternary Bonsai. This repository contains only the OpenVINO conversion by Wondernutts. See the original model card for the authoritative description and intended use.

Important: this is not the native ternary build

The source checkpoint expands the ternary model into ordinary FP16 tensors for compatibility. This repository applies conventional OpenVINO INT4 compression to those unpacked tensors; it does not use PrismML's native ternary packing or custom kernels.

For the efficient native deployment, use PrismML's ternary MLX or ternary GGUF release.

Conversion details

The repository includes language, text-embedding, vision, tokenizer, and detokenizer IRs. The source configuration's maximum context has not been independently validated for this conversion.

Download

Download the public repository:

python -m pip install "openvino-genai==2026.2.1.0" "huggingface_hub>=1.0"
hf download Wondernutts/Ternary-Bonsai-27B-int4-sym-ov --local-dir ./ternary-bonsai-27b-int4-ov

Run on Intel GPU

import openvino_genai as ov_genai

model_dir = "./ternary-bonsai-27b-int4-ov"
pipe = ov_genai.VLMPipeline(model_dir, "GPU", CACHE_DIR="./ov_cache")

config = ov_genai.GenerationConfig()
config.max_new_tokens = 512
config.do_sample = True
config.temperature = 0.8
config.top_p = 0.95

pipe.start_chat("You are a thoughtful and precise assistant.")
print(pipe.generate("Write a short scene set in a city floating above the clouds.", generation_config=config))
pipe.finish_chat()

The shipped tokenizer applies the model's chat template. Change "GPU" to "CPU" for CPU inference. For continuous batching on Intel GPU, set DYNAMIC_QUANTIZATION_GROUP_SIZE=0 if dynamic activation quantization causes garbled output.

For image input, load an RGB image into an openvino.Tensor and pass it as images=[tensor] to VLMPipeline.generate(). Vision and long-context behavior have not been independently benchmarked for this build.

License and attribution

Apache-2.0, following the source repository. Model authorship and training credit belong to PrismML; OpenVINO conversion credit belongs to Wondernutts.