Ternary-Bonsai-2-27B-GGUF-Colab-Prebuilt-GPU

Ternary-Bonsai-2-27B (GGUF) – Fast Colab Inference 🚀

This repository provides prebuilt binaries and GGUF configurations to run Ternary-Bonsai-2-27B on Google Colab or similar GPU environments with minimal setup.
The default PTQ1_0 variant is only ~5.95 GB and can be downloaded + launched in roughly 1–2 minutes on Colab.

---

⚡ Quick Start

🔧 Setup (run once)

!pip install -q huggingface_hub

import os
from huggingface_hub import hf_hub_download

# --- CONFIGURATION ---
repo_id = "IamPradeep/Ternary-Bonsai-2-27B-GGUF-Colab-Prebuilt-GPU"

# ------------------------------------------------------------------------------
# 1. Download + extract the pre-compiled llama.cpp binaries (PrismML fork)
# ------------------------------------------------------------------------------
print("📥 1. Downloading pre-compiled binary package...")
bin_tar_path = hf_hub_download(repo_id=repo_id, filename="llama_bin.tar.gz")

# Extract the binaries into a folder called 'llama_bin'
# Using tar preserves the executable bit, unlike zip.
!mkdir -p ./llama_bin
!tar -xzf {bin_tar_path} -C ./llama_bin

# Grant executable permissions to the main CLI
!chmod +x ./llama_bin/llama-cli

print("✅ Binaries ready at ./llama_bin/")

# ------------------------------------------------------------------------------
# 1b. Install the CUDA 12 runtime libraries required by the pre‑built binary.
# ------------------------------------------------------------------------------
print("📥 Installing CUDA 12 runtime libraries...")
!pip install -q nvidia-cuda-runtime-cu12 nvidia-cublas-cu12

# ------------------------------------------------------------------------------
# 2. Download the PTQ1_0 model (dense trits, ~5.95 GB)
# ------------------------------------------------------------------------------
print("\n\n📥 2. Downloading PTQ1_0 model (this takes ~1 minute for ~5.95 GB on Colab)...")
model_path = hf_hub_download(
    repo_id=repo_id,
    filename="Ternary-Bonsai-2-27B-PTQ1_0.gguf",
)

# ------------------------------------------------------------------------------
# 3. Environment setup — put the extracted dir AND the CUDA pip‑installed
#    libraries on LD_LIBRARY_PATH so the loader can find libcudart.so.12.
# ------------------------------------------------------------------------------
import site
import os

# The nvidia-* pip packages install their shared libraries under:
# <site-packages>/nvidia/<component>/lib
nvidia_lib_dirs = []
for pkg in ("cuda_runtime", "cublas"):
    pkg_path = os.path.join(site.getsitepackages()[0], "nvidia", pkg, "lib")
    if os.path.isdir(pkg_path):
        nvidia_lib_dirs.append(pkg_path)

# Build the final LD_LIBRARY_PATH: ./llama_bin + NVIDIA libs + existing path
existing = os.environ.get("LD_LIBRARY_PATH", "")
parts = ["./llama_bin"] + nvidia_lib_dirs
if existing:
    parts.append(existing)
os.environ["LD_LIBRARY_PATH"] = ":".join(parts)

---

🤖 Text Inference

system_prompt = "You are a helpful AI assistant."
prompt        = "Explain quantum computing in simple terms."

print("\n--- Running inference with Ternary-Bonsai-2-27B (PTQ1_0) ---\n")

!./llama_bin/llama-cli \
  -m "{model_path}" \
  -ngl 99 \
  -c 8192 \
  -sys "{system_prompt}" \
  -p "{prompt}" \
  --temp 0.7 \
  --top-p 0.95 \
  -n 2048 \
  --reasoning off  # Use "on" to enable reasoning/thinking

---

👁️ Vision / Multimodal Inference (Live Upload)

You can run vision tasks by downloading a vision projector (mmproj) and uploading an image directly inside Google Colab:

from PIL import Image
from google.colab import files
from huggingface_hub import hf_hub_download

# 1. Download Vision Projector (~629 MB)
print("Downloading Vision Projector...")
mmproj_path = hf_hub_download(
    repo_id=repo_id,
    filename="Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf",
)

# 2. Live Image Upload
print("\n" + "="*50)
print("📸 PLEASE UPLOAD AN IMAGE FROM YOUR DEVICE:")
print("="*50)

uploaded = files.upload()

if not uploaded:
    print("\n❌ No file was uploaded!")
else:
    # Save the uploaded file path
    image_filename = list(uploaded.keys())[0]
    image_path = f"./{image_filename}"

    print(f"\n✅ Uploaded successfully: {image_filename}")
    display(Image.open(image_path))

    # 3. Run Multimodal Inference
    system_prompt = "You are an expert AI vision assistant."
    prompt = "Describe what you see in this image in detail."

    print(f"\n🤖 Running Vision Inference on {image_filename}...\n")

    !./llama_bin/llama-cli \
      -m "{model_path}" \
      --mmproj "{mmproj_path}" \
      --image "{image_path}" \
      -sys "{system_prompt}" \
      -p "{prompt}" \
      -ngl 99 \
      -c 8192 \
      --temp 0.2 \
      --top-p 0.95 \
      -n 2048 \
      --reasoning off   # Use "on" to enable reasoning/thinking

---

💡 Notes

• Default model: Ternary-Bonsai-2-27B-PTQ1_0.gguf is ~5.95 GB, using dense ternary trits for fast GPU inference with a small VRAM footprint.
• Prebuilt binaries: llama_bin.tar.gz is ~35.2 MB and contains the PrismML fork of llama.cpp. No build or compile step is needed.
• GPU acceleration: -ngl 99 offloads all layers to the GPU. Use a Colab GPU runtime for best results.
• Reasoning mode: Change --reasoning off to --reasoning on to enable reasoning/thinking.
• Vision projector (default): Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf (~629 MB)
• Vision projector (high fidelity): Ternary-Bonsai-2-27B-mmproj-BF16.gguf (~931 MB)
• Other model variants: Simply change the filename parameter in hf_hub_download.

---

📦 Available Files

---

🧠 Architecture

Ternary-Bonsai-2-27B is part of the Bonsai-2 family, using 1.58-bit ternary quantization with weights represented as {-1, 0, +1}.
This allows a 27B parameter model to run in a compact 5.95 GB footprint while retaining strong reasoning and multimodal capability when paired with an mmproj vision projector.

The PTQ1_0 variant uses dense trits and is the recommended default for Colab and low-VRAM GPU environments.

---

🤝 Acknowledgments

Special thanks to PrismML, the original architecture creators behind the Bonsai model families. Their ternary / 1.58-bit quantization methods make high-capability 27B local inference possible in a very small GPU footprint.