Ternary-Bonsai-27B-Grug-LoRA-GGUF

<p align="center"> <img src="./assets/cave-gardener-bonsai.png" alt="A cave gardener carefully trimming a tiny bonsai tree" width="100%"> </p> <h1 align="center">Grug LoRA for Ternary Bonsai · GGUF</h1> <p align="center"> A compact rank-64 adapter that gives compatible Qwen3.6-27B models the terse, token-conscious Grug reasoning style. </p> <p align="center"> <a href="https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf">Ternary Bonsai base</a> · <a href="https://huggingface.co/ProCreations/grug-27b">Original Grug model</a> · <a href="https://huggingface.co/signsur4739379373/Qwen3.6-27B-Grug-LoRA">Rank-64 source adapter</a> </p>

---

Pick a build

<table> <tr> <td width="33%" valign="top"> <strong>Q8_0 · Recommended</strong><br><br> Best balance of quality, speed, and memory. Fastest single-request result in our tests.<br><br> <strong>323 MiB</strong><br> <a href="https://huggingface.co/ajh-code/Ternary-Bonsai-27B-Grug-LoRA-GGUF/resolve/main/Grug-LoRA-r64-Q8_0.gguf?download=true">Download Q8_0</a> </td> <td width="33%" valign="top"> <strong>F16 · Highest fidelity</strong><br><br> Closest practical copy of the source adapter. Best measured DSpark and batched throughput.<br><br> <strong>608 MiB</strong><br> <a href="https://huggingface.co/ajh-code/Ternary-Bonsai-27B-Grug-LoRA-GGUF/resolve/main/Grug-LoRA-r64-F16.gguf?download=true">Download F16</a> </td> <td width="33%" valign="top"> <strong>NVFP4 · Experimental</strong><br><br> Smallest build, but lower fidelity and a custom CUDA backend are required. Not for stock llama.cpp.<br><br> <strong>171 MiB</strong><br> <a href="https://huggingface.co/ajh-code/Ternary-Bonsai-27B-Grug-LoRA-GGUF/resolve/main/Grug-LoRA-r64-NVFP4.gguf?download=true">Download NVFP4</a> </td> </tr> </table>

This repository contains LoRA adapters only. Download a compatible Qwen3.6-27B base model separately. Testing used Ternary-Bonsai-27B-Q2_0.gguf.

---

Quick start

hf download prism-ml/Ternary-Bonsai-27B-gguf \
  Ternary-Bonsai-27B-Q2_0.gguf --local-dir .

hf download ajh-code/Ternary-Bonsai-27B-Grug-LoRA-GGUF \
  Grug-LoRA-r64-Q8_0.gguf --local-dir .

llama-server \
  -m Ternary-Bonsai-27B-Q2_0.gguf \
  --lora-scaled Grug-LoRA-r64-Q8_0.gguf:1.5 \
  -ngl 99 -c 32768

Scale 1.5 is recommended for Ternary Bonsai. It compensates for the changed ternary base and produced the best completion reliability and token efficiency in our calibration. Scale 1.0 matches the source-adapter baseline, but was too weak on this base.

---

Measured on RTX 5080

These are adapter comparisons, not universal speed claims. DSpark used a separate RTX 5060 Ti drafter; acceptance rate strongly affects its speed.

---

Compatibility and limits

• F16 and Q8_0 use standard GGUF LoRA tensor formats.
• NVFP4 is a research artifact for a custom Blackwell CUDA path and is not compatible with stock llama.cpp. It is included for research and archival comparison while the companion backend remains unpublished.
• Short quality checks passed in all three formats. Q8_0 is the recommended release; NVFP4 showed a long-form completion failure and remains experimental.
• Apache-2.0. This conversion is derived from the linked Grug projects and is tested with, but not created by or affiliated with, Prism ML.

<details> <summary><strong>What was tested?</strong></summary>

Single-stream decoding, split-GPU DSpark, real concurrent batches through llama-server, deterministic behavior checks, and two preallocated 134,144-token KVarN K4/V4 slots. Full methods and raw results are maintained in the companion research project.

</details>