Ternary-Bonsai-8B-ROCm-gfx1151

Ternary-Bonsai-8B on ROCm (gfx1151) — which of the four files to actually use

This repository contains no weights. It measures prism-ml/Ternary-Bonsai-8B-gguf on AMD ROCm, which its model card does not cover.

That repo ships three low-bit files (plus an F16 source), and a community TQ2_0 conversion exists as well. The four are not equivalent: the spread is 2.2x in speed, one of them will not load at all in a current build, and the fastest one is also the smallest.

Measured on AMD Strix Halo (Radeon 8060S, gfx1151), ROCm 7.13, PrismML fork prism @ 9a9394a.

Results

Three distinct generation tasks and three distinct verbatim-rewrite sources per arm, fresh server per arm, cache_prompt:false. All loading arms returned correct output 3/3.

Use PQ2_0. It is simultaneously the smallest file and the fastest — 2.23x the throughput of the standard-ggml TQ2_0 conversion, which is also 22% larger. Q2_0_g64 is a close second if you need a non-PQ format.

⛔ Ternary-Bonsai-8B-Q2_0.gguf does not load in a current build

gguf_init_from_reader: this file matches the legacy Prism Q2_0 layout
    (group size 128 stored as ggml type id 42),
    but this build reads Q2_0 as the official group-64 format

It is a legacy g128 file. Use PQ2_0, or Q2_0_g64.gguf from the same repo. The error message is explicit, but the filename gives no hint, so it is easy to download the wrong one.

The standard ggml ternary type is the slow path here

TQ2_0 (ggml type 35) is the portable option — it loads in any recent llama.cpp, including forks that know nothing about PrismML's private types. We measured it at 34.88 t/s against PQ2_0's 77.87 t/s on the same binary, so the gap is the format, not the runtime. Confirmed by running the same TQ2_0 file on an unrelated ROCm fork: 33.75 t/s — 3.2% off the Prism build.

⇒ Portability costs you more than half the throughput on this model. If you can run PrismML's fork, use PQ2_0.

This model is much simpler than Ternary-Bonsai-2 27B

No prism.hadamard.* metadata means no rotated basis, no inverse transform, and no source patch. Declared context is 65536 and it serves in full. Tool calling works with --jinja. Sampling defaults carried in the GGUF: temp 0.5, top_k 20, top_p 0.85, min_p 0.0.

Files

Reproduction

# runtime: PrismML's fork built for gfx1151
git clone -b prism https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp   # tested at 9a9394a
export ROCM_PATH=/opt/rocm HIP_PATH=/opt/rocm
cmake -S . -B build-rocm -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DAMDGPU_TARGETS=gfx1151 \
  -DCMAKE_HIP_COMPILER=/opt/rocm/lib/llvm/bin/clang++ \
  -DCMAKE_HIP_COMPILER_ROCM_ROOT=/opt/rocm \
  -DGGML_HIP_GRAPHS=ON -DGGML_HIP_NO_VMM=ON -DLLAMA_CURL=OFF -DLLAMA_BUILD_WEBUI=OFF
cmake --build build-rocm -j8 --target llama-server

hf download prism-ml/Ternary-Bonsai-8B-gguf Ternary-Bonsai-8B-PQ2_0.gguf --local-dir .

./build-rocm/bin/llama-server -m Ternary-Bonsai-8B-PQ2_0.gguf \
  -dev ROCm0 -ngl 999 -fa on --no-mmap -fit off \
  -np 1 -b 2048 -ub 1024 -t 16 -c 65536 --jinja

⛔ Plain -DGGML_HIP=ON fails with "Failed to find ROCm root directory" — pass CMAKE_HIP_COMPILER explicitly.

Credits and licence

Weights by prism-ml; the TQ2_0 conversion measured here is ewchampion/Ternary-Bonsai-8B-TQ2_0-GGUF. This repository adds only measurements. Apache-2.0, matching upstream.