Ternary-Bonsai-8B-GGUF-Colab-Prebuilt-GPU

Ternary-Bonsai-8B (GGUF) – Run in 30 Seconds 🚀

This repository provides optimized pre-built binaries and configurations to get the Ternary-Bonsai-8B model up and running on Google Colab or similar environments with GPU acceleration in under 30 seconds.

⚡ Quick Start

🔧 Setup (run once)

!pip install -q huggingface_hub

import os
import site
from huggingface_hub import hf_hub_download

# --- CONFIGURATION ---
repo_id = "IamPradeep/Ternary-Bonsai-8B-GGUF-Colab-Prebuilt-GPU"

print("1. Downloading pre-compiled binary...")
bin_zip_path = hf_hub_download(repo_id=repo_id, filename="llama_bin.tar.gz")

# Extract the binaries into a folder called 'llama_bin'
!mkdir -p ./llama_bin
!tar -xzvf {bin_zip_path} -C ./llama_bin
# Ensure Linux knows it is an executable
!chmod +x ./llama_bin/llama-cli

# ------------------------------------------------------------------------------
# 1b. Install the CUDA 12 runtime libraries required by the pre-built binary
# ------------------------------------------------------------------------------
print("📥 Installing CUDA 12 runtime libraries...")
!pip install -q nvidia-cuda-runtime-cu12 nvidia-cublas-cu12 nvidia-cuda-nvrtc-cu12

print("\n2. Downloading model (this takes ~15 seconds)...")
model_path = hf_hub_download(repo_id=repo_id, filename="Ternary-Bonsai-8B-Q2_0.gguf")

# ------------------------------------------------------------------------------
# 3. Environment setup — put the extracted dir AND the pip‑installed CUDA
#    libraries on LD_LIBRARY_PATH so the loader can find libcudart.so.12.
# ------------------------------------------------------------------------------
nvidia_lib_dirs = []
for pkg in ("cuda_runtime", "cublas", "cuda_nvrtc"):
    pkg_path = os.path.join(site.getsitepackages()[0], "nvidia", pkg, "lib")
    if os.path.isdir(pkg_path):
        nvidia_lib_dirs.append(pkg_path)

existing = os.environ.get("LD_LIBRARY_PATH", "")
parts = ["./llama_bin"] + nvidia_lib_dirs
if existing:
    parts.append(existing)
os.environ["LD_LIBRARY_PATH"] = ":".join(parts)

---

🤖 Run Inference

# ------------------------------------------------------------------------------
# 4. Run the model
# ------------------------------------------------------------------------------
system_prompt = "You are a helpful assistant."
prompt = "Explain quantum computing in simple terms."

print(f"\n--- Running inference ---\n")

!./llama_bin/llama-cli \
  -m "{model_path}" \
  -ngl 99 \
  -c 8192 \
  -sys "{system_prompt}" \
  -p "{prompt}" \
  --temp 0.7 \
  --top-p 0.95 \
  -n 2048

---

🤝 Acknowledgments

Special thanks to PrismML, the original architecture creators behind the revolutionary 1-bit and ternary (1.58-bit) Bonsai model families. Their custom quantization methods allow this 8B parameter model to deliver rapid local inference inside an ultra-compact ~2 GB footprint.