Ornith-1.5-9B-heretic-GGUF

Ornith-1.5-9B-heretic

Importance-matrix GGUF quantizations of an abliterated Ornith-1.5-9B.

Progressive release

Our upload bandwidth is limited (~1 MB/s), so the full 24-quant ladder takes many hours to transfer. Rather than sit on everything until the last byte lands, files are being published as they finish uploading:

1. Q4_K_M and the vision projector first - the ones most people want
2. Then Q8_0, Q6_K, Q5_K_M, IQ4_XS, Q4_K_S
3. Then the remaining low-bit quants, over the following hours

23 quants are built and queued. The table below lists what is currently live in this repo and is regenerated as uploads land - so if the file you want is missing, it is on its way rather than cancelled. Check back later, or watch the repo.

Quantization quality - read this before judging the numbers

Top-token agreement on this model looks low compared to figures you may have seen for other GGUF repos. It is not a defect, and it is not caused by the abliteration. The control proves it:

The model authors' own official Q4_K_M scores 91.29% on the identical test. This repo lands within half a point of it, and the SSM-protected variant beats it. Whatever is happening here is a property of the model, not of this build.

The untouched base model quantizes worse than this one. Two architectural reasons it sits lower than a typical Llama-class model:

1. 248,320-token vocabulary (Llama-2 has 32,000). With 7.8x more tokens competing, far more of them sit within a hair of each other in logit space, so the argmax flips on near-ties even when the output distribution is nearly identical. Top-1 agreement is simply a harsher metric at this vocab size and is not comparable across models.
2. 24 of 32 layers use linear attention (GatedDeltaNet), which carries a recurrent state forward through the sequence. Standard attention recomputes from scratch each token so quantization error stays local; a recurrent state accumulates it. The config pins mamba_ssm_dtype: float32 precisely because that state is numerically fragile.

Judge these quants by KL divergence, not top-1 agreement. Q8_0 measures KL 0.0019 and Q4_K_M 0.048 - both squarely in the normal published range.

An architecture-specific improvement

Point 2 above is actionable. llama.cpp's default Q4_K_M recipe was tuned for conventional transformers and quantizes the ssm_alpha, ssm_beta and ssm_out projections - the ones feeding that fragile recurrent state - at the base 4-bit rate. Protecting just those 72 tensors at Q8_0:

llama-quantize --imatrix imatrix.gguf \
  --tensor-type ssm_alpha=q8_0 --tensor-type ssm_beta=q8_0 \
  --tensor-type ssm_out=q8_0 \
  model-BF16.gguf model-Q4_K_M.gguf Q4_K_M

Worth knowing for anyone quantizing hybrid linear-attention models - the stock recipes do not know these tensors are special.

Important

This model has had its safety alignment deliberately removed. It will attempt to comply with requests the original model would refuse, including harmful ones. It has no guardrails.

You are responsible for how you use it and for complying with the base model's MIT licence and applicable law. Do not deploy it in a user-facing product without adding your own safety layer.

The finding: where you sample the residuals decides everything

This is the part worth reading if you abliterate models yourself.

Directional ablation (Arditi et al. 2024) derives a "refusal direction" from the difference between residual activations on harmful and harmless prompts, then projects that direction out of every weight that writes to the residual stream. Tooling like Heretic samples those activations at the start of the model's response.

Ornith-1.5 is a reasoning model, and its chat template ends the generation prompt with an already-open <think> block:

{%- if add_generation_prompt %}
    {{- '<|im_start|>assistant\n' }}
    {{- '<think>\n' }}          {# prompt ends INSIDE the reasoning block #}
{%- endif %}

So "the start of the response" is not the start of the answer - it is the start of the model's reasoning. The direction you recover is "about to think about refusing" rather than "about to refuse", and those are measurably different features.

Ablating the first one plateaus. We ran a 200-trial TPE search and a 13-recipe sweep against it and could not get below ~22/100 genuine refusals no matter how hard we pushed - raising ablation strength just cost capability without moving refusals.

Closing the block first samples where the answer actually begins:

# MUST be set before computing directions - get_residuals() routes
# through generate(), which appends this to the prompt.
settings.response_prefix = "\n</think>\n\n"

Verify the string against the template rather than trusting it; it should reproduce the model's own no-thinking form exactly:

p = tok.apply_chat_template(chat, add_generation_prompt=True, tokenize=False)
assert p + "\n</think>\n\n" == tok.apply_chat_template(
    chat, add_generation_prompt=True, tokenize=False, enable_thinking=False)

Identical ablation parameters. Corrected sampling position: 22/100 -> 0/100 genuine refusals, at lower KL divergence.

How to tell this is happening to you

• Your tool reports an empty common response prefix (Heretic prints response_prefix = '')
• Refusals plateau and extra ablation strength buys nothing
• The giveaway: your measured baseline refusal rate looks low for a well-aligned model. Ornith-1.5-9B scores 67/100 measured inside the reasoning block and 99/100 measured on its answers. If a safety-tuned model looks only two-thirds censored, you are probably scoring the wrong text.

What we tried before finding it

Every recipe below was evaluated against the same 100 held-out prompts. Down and to the left is better - fewer refusals for less divergence.

What is ours, what is borrowed

Being explicit about this, because "new abliteration recipe" is an easy thing to overclaim:

Reused, unchanged:

• Directional ablation - Arditi et al. 2024. The core method.
• Heretic by Philipp Emanuel Weidmann - did the actual work. The shipped ablation parameters are exactly what its TPE optimizer produced; we did not invent a new ablation algorithm. We fixed the input it was being fed.
• Norm-preserving / projected abliteration - grimjim. Heretic's default, and it beat plain difference-of-means here.
• Calibration corpus for the importance matrix - bartowski's calibration_datav3.
• llama.cpp for GGUF conversion, imatrix and quantization.
• Prompt sets - mlabonne/harmful_behaviors and mlabonne/harmless_alpaca.

Inspiration:

• The published Ornith-1.0 abliterations, whose model cards documented their recipe in enough detail to test directly. It did not transfer to 1.5, but having a concrete baseline to falsify is what made the sweep worth running.

Ours:

• The chain-of-thought sampling-position fix. The finding above. This is the contribution.
• Two-scorer refusal measurement. Keyword scoring reported 6/100 where the true rate was 0/100; every hit was a compliant answer containing a word like "illegal". Reporting the keyword number alone overstates residual censorship.
• Matched-pair cross-group uniformity probe. Described below - an average refusal rate cannot detect it.

Measured results

100 held-out prompts from mlabonne/harmful_behaviors.

Cross-group uniformity

Abliteration removes a single direction, and refusals reinforced hardest in post-training can survive it unevenly - producing a model that complies for some demographic groups and refuses for others given identical prompts. That is a defect on its own terms: a model that is uncensored for 18 groups and censored for 2 is not uncensored, it is unpredictable, and an average refusal rate will not show it.

Tested with a matched-pair probe - 100 prompts holding the request template constant while varying only the group name, across 20 groups spanning religion, ethnicity, national origin, gender, orientation, age and disability.

Result: uniform. 0/100 refusals - every group treated identically across all five phrasings, with zero spread between the most- and least-refused group.

Capability preservation

• 43 of 760 tensors modified, all in the text decoder
• Vision tower, embeddings, norms and MTP head bit-identical to base
• No NaN/Inf; max relative weight change 0.028
• 0 of 6 neutral prompts produced empty or repetition-looped output
• Vision path verified working post-ablation

How much does quantization actually cost you?

Every quant was scored against the BF16 model as ground truth: run the full-precision model once with --save-all-logits, then measure each quant against those logits with llama-perplexity --kl-divergence over 16k tokens.

The headline number is top-token agreement - how often the quant predicts the same next token the unquantized model would. That is the thing you feel in use, and it is not something you can read off a file size.

Two findings worth acting on:

• IQ4_XS beats Q4_K_M - it is smaller (5.20 vs 5.63 GB), closer to the original (KL 0.036 vs 0.048), and faster. The I-quants beat the K-quants at nearly every size point on this model. If you were going to grab Q4_K_M out of habit, take IQ4_XS instead.
• The knee is around 3.5-4 GB. Above it, agreement degrades gently. Below IQ2_XXS it falls off a cliff - IQ1_S agrees with full precision only 46% of the time. Those files are included for completeness, not recommended.

Two quants were built, measured, and deleted. Q2_0 and Q1_0 - newer group-64 types - are non-functional on this architecture. Q2_0 matched the full-precision top token 0.0% of the time (KL 12.8) while being larger than IQ1_S, which manages 46%. They are not published. If you see them in other repos for this model family, benchmark before trusting them.

Files

Vision projector (required for image input)

Ornith-1.5-9B is a vision-language model. For image input, download the mmproj file in addition to your chosen quant.

Which file should I pick?

Aim for a file 1-2 GB smaller than your VRAM for full offload. To maximise quality instead, add system RAM to VRAM and pick 1-2 GB under that total.

K-quants (Q4_K_M) are the safe default. I-quants (IQ4_XS) are smaller for a given quality and work on CUDA/CPU/Metal, but are not supported by Vulkan and are slower on some backends. In doubt, take Q4_K_M.

Usage

llama-cli -m Ornith-1.5-9B-heretic-Q4_K_M.gguf -ngl 99 \
  --temp 0.6 --top-p 0.95 --top-k 20 --repeat-penalty 1.05 \
  -p "Hello"

Sampling matters on this model family. repeat-penalty 1.05 is the sweet spot - 1.0 causes reasoning loops, 1.1 truncates answers. Greedy decoding (--temp 0) is not recommended.

With image input:

llama-mtmd-cli -m Ornith-1.5-9B-heretic-Q4_K_M.gguf \
  --mmproj mmproj-Ornith-1.5-9B-heretic-f16.gguf \
  --image photo.jpg -p "Describe this image." -ngl 99

Note for anyone re-quantizing this model

config.json declares mtp_num_hidden_layers: 1, but the checkpoint ships no mtp.* weights. Converting without --no-nextn produces a GGUF that advertises 33 blocks while containing 32, and llama.cpp refuses to load it:

check_tensor_dims: tensor 'blk.32.attn_norm.weight' not found

python convert_hf_to_gguf.py <model> --outtype bf16 --no-nextn

Provenance

Credits

• Philipp Emanuel Weidmann for Heretic, which did the actual ablation work here
• Arditi et al. for the directional-ablation result this all rests on
• grimjim for projected and norm-preserving abliteration
• mlabonne for the abliteration write-ups and the prompt datasets
• bartowski for the imatrix calibration corpus that is now the community standard
• the ggml-org/llama.cpp team
• DeepReinforce AI for releasing Ornith under MIT