Downloads · 30 days
3K
53% of all-time downloads
Override-6/Ternary-Bonsai-2-27B-abliterated-gguf
Ternary-Bonsai-2-27B-abliterated-gguf is a text generation model from Override-6. Use it when you need the model to write or continue text. It is set up for llama.cpp. The card lists the license as apache-2.0.
This is an abliterated version of prism-ml/Ternary-Bonsai-2-27B-gguf (PTQ10 packing). It keeps the exact original file size, 5,946,648,928 bytes: - Every tensor has its original type and shape. - All 66 edited weight…
Downloads · 30 days
3K
53% of all-time downloads
All-time downloads
5.6K
Public
Repo size
5.9 GB
Likes
2
Public
Click a slice to open those files.
.gguf5.9 GB · 100%
From the Hugging Face model README
This is an abliterated version of prism-ml/Ternary-Bonsai-2-27B-gguf (PTQ1_0 packing). It keeps the exact original file size, 5,946,648,928 bytes:
{-1, 0, +1} × fp16 group-128 scales.Created using Bonsai by Prism ML. Built from Qwen3.8-27B (Alibaba Cloud). See
NOTICE.txt.
These files need the PrismML llama.cpp fork, exactly like the original. Stock llama.cpp cannot run PTQ1_0 or apply the Hadamard activation transform.
# https://github.com/PrismML-Eng/llama.cpp/releases (tested with prism-b10709-9a9394a, CUDA 12.4)
./llama-server -m Ternary-Bonsai-2-27B-PTQ1_0-abliterated.gguf -ngl 99 -fa on -c 32768
Sampling is the same as the base model: thinking mode temp 1.0, top_p 0.95, top_k 20.
All numbers were measured locally on an RTX 4070 with the fork above.
| Original PTQ1_0 | Naive abliteration (writers re-quantized to Q6_K) | This model | |
|---|---|---|---|
| File size | 5.95 GB | 9.36 GB | 5.95 GB |
| Harmful prompts, thinking mode: complied / deflected / refused | 0 / 3 / 101 | – | 76 / 27 / 1 |
| Harmful prompts, no-think mode: complied / deflected / refused / empty | 1 / 8 / 95 / 0 | 44 / 50 / 10 / 0 | 74 / 30 / 0 / 0 |
| MMLU, log-prob scoring (n=1000) | 78.9 | 78.2 | 79.0 |
| GSM8K, thinking mode (n=100) | 94.0 | 93.0 | 94.0 |
| IFEval, prompt-level strict (n=541, no-think) | 78.4 | 78.0 | 81.7 |
| IFEval, instruction-level strict | 85.5 | 84.7 | 87.1 |
| Needle-in-a-haystack, 8k / 16k / 32k / 64k (n=40) | 100 / 100 / 100 / 100 | 100 / 100 / 100 / 100 | 100 / 100 / 100 / 100 |
| KL divergence vs original, harmless chats | 0 | 0.0240 | 0.0189 |
| KL divergence vs original, wikitext-2 | 0 | 0.0134 | 0.0130 |
Evaluation setup
mlabonne/harmful_behaviors (test split). Responses
are labeled by an LLM judge (the original model in non-thinking mode, judge.py):
llama-perplexity --kl-divergence against the original model. The
harmless chat corpus is the original model's own answers to 200 held-out harmless_alpaca prompts.lm_eval). Needle-in-a-haystack hides a 7-digit
code at 5 depths inside wikitext contexts.Other community builds were evaluated with the same harness: same held-out prompts, same judge, same KL corpora, same flags. Numbers can differ from the ones on their own cards, which use other prompt sets and judges.
| Size | No-think: complied / deflected / refused | Thinking: complied / deflected / empty | KL chat / wiki | MMLU | GSM8K | IFEval strict | NIAH 8k–64k | |
|---|---|---|---|---|---|---|---|---|
| Original (prism-ml) | 5.95 GB | 1 / 8 / 95 | 0 / 3 / 0 | 0 / 0 | 78.9 | 94 | 78.4 | 100% |
| This model | 5.95 GB | 74 / 30 / 0 | 76 / 27 / 0 | 0.019 / 0.013 | 79.0 | 94 | 81.7 | 100% |
| OS-Software Heretic (PTQ1_0) | 5.95 GB | 97 / 4 / 3 | 101 / 0 / 3 | 0.011 / 0.008 | 77.3 | 91 | 78.2 | 100% |
| dealignai CRACK (PQ2_0) | 7.21 GB | 88 / 16 / 0 | 98 / 5 / 1 | 0.046 / 0.060 | 78.9 | 93 | – | – |
| BoldingBuilds (PTQ1_0) | 5.95 GB | 47 / 57 / 0 | 56 / 29 / 19 | 0.055 / 0.050 | 79.5 | 92 | – | – |
Summary: the OS-Software Heretic build is more uncensored: it deflects far less, with lower KL. This model keeps every capability benchmark at or above the original.
Prior work on lattice editing: editing the ternary/2-bit codes directly, instead of re-quantizing, is not unique to this model. BoldingBuilds, Hikari07jp and OS-Software released lattice-edited builds earlier. What this model adds is the published end-to-end pipeline and the scale-refit healing step.
The scripts are in scripts/. The fixed refusal direction is scripts/directions/dirs_proj_both.npy.
Abliteration replaces every residual-stream writer W with W − α·r·rᵀW. On this model:
make_dirs.py, tools/actdump.cpp)l_out) of every layer at the last prompt token, for 400 harmful and
400 harmless training prompts.<think>\n (thinking mode) and </think>\n\n (no-think
mode). The two per-mode directions only have cosine 0.64. A direction taken from one mode only
leaves the other mode broken: the model returns empty answers.abliterate.py)The trits are edited directly. The PTQ1_0 block codec is re-implemented and verified to round-trip byte-exact.
For each writer in layers 16–48, the exact delta −α·r·(rᵀW) is realized by unbiased
stochastic trit flips, followed by a small greedy correction so that rᵀW′ = (1−α)·rᵀW holds
column by column.
Per-component strength:
attn_output, ssm_out)ffn_downMLP edits cost far more KL per unit of refusal removed, consistent with Heretic's findings.
The Hadamard rotation of this model is on the input axis only, so an output-side projection is exact in the stored basis.
heal.py, tools/wdump.cpp)Capture the rotated inputs of every writer (H = E[xxᵀ]) on 16k calibration tokens:
None of these overlap the evaluation sets.
For each edited writer, refit only the fp16 group scales by ridge least squares, per output row:
min_s (s·B − w*) H (s·B − w*)ᵀ, where w* = (I − α·r·rᵀ)W is the exact abliterated target.
Trits, layout and size are unchanged.
| Approach | Result |
|---|---|
| Raw (unprojected) direction on all layers | KL 1.4–7.8, model broken |
| Direction from the thinking position only | many empty answers in no-think mode |
| Narrower layer bands (16–40, 16–44) | refusal comes back (Hydra effect) |
| GPTQ trit re-selection | output error is lower, but it does not remove the direction by itself |
| Two orthogonal directions | worse on both refusal and KL |
| Overshoot α = 1.3–1.5 on all writers | 0% refusal but KL 2–4× higher and many empty answers |
actdump on thinking and no-think prompt files (prompts.py), then make_dirs.py.
wdump on the calibration corpus (gen_calib.py), all layers.
Build the model:
python abliterate.py --src Ternary-Bonsai-2-27B-PTQ1_0.gguf --out cand.gguf \
--dirs directions/dirs_proj_both.npy --dir-layer 36 --layers 16-48 --alpha-attn 2.0 --alpha-ffn 0.7
python heal.py --orig Ternary-Bonsai-2-27B-PTQ1_0.gguf --edited cand.gguf --out final.gguf --acts wacts \
--dirs directions/dirs_proj_both.npy --dir-layer 36 --layers 16-48 --alpha-attn 2.0 --alpha-ffn 0.7 --recorrect 0
Evaluate with evalmodel.sh (refusal + KL), bench.py, mmlu_lp.py and judge.py.
Stochastic flips use seed 0.
This model has had its refusal behavior removed. It will produce content the original model would decline to produce. You are responsible for how you use it and for complying with applicable laws.