Ornith-1.5-9B-GGUF

How to Run Ornith 1.5 9B Locally

<p style="margin-top: 0; margin-bottom: 0;"> <em>Built from Ornith's original weights with our own importance matrix. The <a href="https://huggingface.co/datasets/AtomicChat/calib-corpora">calibration corpora</a> and <a href="https://huggingface.co/datasets/AtomicChat/Ornith-1.5-9B-GGUF-metrics">raw eval logs</a> behind this card are public.</em> </p> <div style="display: flex; gap: 8px; align-items: center; margin-top: 10px; margin-bottom: 10px;"> <a href="https://atomic.chat/?utm_source=huggingface&utm_medium=referral&utm_campaign=hf_ornith_1_5_9b&utm_content=btn_atomic"><img src="https://huggingface.co/AtomicChat/Ornith-1.5-9B-GGUF/resolve/main/btn_atomic.png" width="162" alt="Atomic Chat"></a> <a href="https://discord.gg/8wGSsvmg4V"><img src="https://huggingface.co/AtomicChat/Ornith-1.5-9B-GGUF/resolve/main/btn_discord.png" width="119" alt="Discord"></a> <a href="https://github.com/AtomicBot-ai/Atomic-Chat"><img src="https://huggingface.co/AtomicChat/Ornith-1.5-9B-GGUF/resolve/main/btn_github.png" width="115" alt="GitHub"></a> </div> <ul style="margin: 0 0 12px 0;"> <li>Learn how to run Ornith 1.5 9B locally - read <a href="https://atomic.chat/blog/guides/how-to-run-ornith-1-5-locally?utm_source=huggingface&utm_medium=referral&utm_campaign=hf_ornith_1_5_9b&utm_content=bullet_guide">our guide</a>.</li> <li>You can now run Ornith 1.5 in <a href="https://atomic.chat/?utm_source=huggingface&utm_medium=referral&utm_campaign=hf_ornith_1_5_9b&utm_content=bullet_app">Atomic Chat</a> with toggles for thinking.</li> <li>See our quantization analysis below for measurements and instructions.</li> </ul> <hr style="margin: 0 0 16px 0;">

Ornith-1.5-9B-GGUF

GGUF builds of ornith-ai/Ornith-1.5-9B, converted and measured by AtomicChat.

Which file to take

Sizes below already include the 0.9 GB vision projector and about 1 GB of runtime overhead at 8k context. Drop the projector if you only want text.

[!WARNING] The two smallest files are published for completeness, not because we recommend them. AD-IQ2_XXS-IQ1_M agrees with the reference on 53.7% of tokens, which means roughly every second token is a different word than the full model would have produced. That is not a small quality drop, it is a different model. If a 9B does not fit, a smaller model at higher precision will serve you better.

Files and measurements

Mean KL divergence against the BF16 reference, held-out eval_neutral corpus, 4096 context, single RTX 5090, llama.cpp b10505. Lower is better. Top-1 is the share of tokens where the quant picks the same next token as BF16.

AD means a per tensor bit layout tuned for this model. Files without the prefix are stock llama.cpp presets built with the same importance matrix, so the comparison isolates the layout and nothing else.

Two results are worth reading off that chart directly.

AD-Q5_K-Q4_K is smaller than stock Q5_K_M and more accurate at the same time, 5.93 GB against 6.47 GB and 0.0255 against 0.0299. No interpolation needed to see it.

At the same size, the tuned layout cuts divergence by about a third. AD-Q4_K-IQ4_XS is 5.61 GB at 0.0344 against stock Q4_K_M at 5.63 GB and 0.0500.

[!NOTE] The tuning only pays while the base type is coarse. Above six bits the stock presets are already close to lossless, so lifting a small part of the model buys nothing while paying for it out of a large part costs real accuracy. We tested this and lost, so above 6 GB this repo ships stock presets rather than a worse tuned file with our label on it.

Quick start

Text only:

llama-server -m Ornith-1.5-9B-AD-Q5_K-Q4_K.gguf \
  -ngl 99 -c 8192 -fa on --jinja \
  --temp 0.6 --top-p 0.95 --top-k 20 \
  --port 8080

With vision:

llama-mtmd-cli -m Ornith-1.5-9B-AD-Q5_K-Q4_K.gguf \
  --mmproj mmproj-Ornith-1.5-9B-F16.gguf \
  -ngl 99 -c 8192 --temp 0.6 --image-min-tokens 1024 \
  --image your-image.jpg \
  -p "Describe this image."

[!IMPORTANT] Pass --image-min-tokens 1024. Without it, dense images such as charts and screenshots get too few visual tokens, and the model answers from what it already knows instead of from what it was shown. llama.cpp prints a warning about this at load time. We hit exactly this on our first attempt: the model produced a confident, fluent, completely invented description of a chart.

[!IMPORTANT] Upstream ships no generation_config.json, so llama.cpp falls back to its own sampling defaults, which are not the ones this model was tuned for. Set them yourself: --temp 0.6 --top-p 0.95 --top-k 20. Use --temp 1.0 to reproduce the benchmark setup from the upstream model card.

Lines reading find_slot: non-consecutive token position are normal. That is how this family numbers image patches, and they appear in successful runs too.

Speed

Single RTX 5090, llama.cpp b10505-ee4c505a4, CUDA 13.0, flash attention on.

Generation is limited by memory bandwidth, not compute: 17.9 GB read per token against 1792 GB/s puts the ceiling near 100 t/s, and BF16 reaches 94% of it. Smaller files scale close to linearly, so a 5.9 GB build lands near 280 t/s on the same card.

Vision

The model reads handwriting. Asked to describe a calligraphy piece it has not seen before, it transcribed the full quote and picked out the small TB-09 mark in the corner.

This image displays an elegant, calligraphic handwritten quote featuring a large blue initial capital "P" and intricate botanical illustrations in the corners. The text reads: "Peace of mind comes to me through making things with my hands. An added bonus comes if my efforts inspire others to try the creative process."

Image encoding takes 163 ms for this picture on one RTX 5090.

Reasoning

A short check that the thinking channel does real work rather than filling space. Prompt: A gas station is 1 kilometer away. I want to wash my car. Should I walk or drive?

The model answered "drive", then argued against itself in the open, noting that some gas stations have car washes so walking there is not obviously wrong, then closed the argument by observing that walking leaves the car at home either way:

You should drive. To wash your car, your car needs to be at the location where it is being washed.

How the layout was chosen

The bit layout was not carried over from another model. We built eight candidate layouts at one size class, measured all of them against the same reference on the same box, and kept the winner. Every log below is in the metrics repo.

Three things came out of this.

The gates matter more than their size suggests. attn_gate and ssm_out are 9% of the weights, and the importance matrix ranks attn_gate first in the whole model by summed squared activation. Lifting those two and paying for them out of the MLP gate and up projections, which are 36% of the weights, is the single largest win available.

The embedding must not be squeezed. With a 248320 token vocabulary and untied weights, the embedding and the output head are 22% of this model, which makes them look like the obvious place to save. AB-T tested that and came last, worse than the stock preset. A lifted output head does not pay for a crushed embedding.

A layout does not transfer by layer position. AB-X applied a recipe that works well on a 64 layer model by scaling the layer numbers, and it lost to the flat control. What transfers is which tensor groups matter, read from the importance matrix of this model.

Architecture

Read from the converted file, not from the config.

Only 8 of the 32 layers keep a KV cache, the other 24 hold a fixed recurrent state instead. Long context is unusually cheap here, roughly 32 KB per token, so 8k of context costs about 256 MB rather than the gigabytes this size of model normally implies.

[!WARNING] If you convert this model yourself, pass --no-nextn to convert_hf_to_gguf.py. Upstream config.json declares mtp_num_hidden_layers: 1, but the checkpoint contains no such tensors. The converter believes the config, writes block_count = 33 for a 32 layer model, and reports a successful export. The file then fails to load with tensor 'blk.32.attn_norm.weight' not found. There is no speculative draft to publish for this model.

Calibration

The importance matrix was computed on the BF16 weights, not on a quantized stand-in, over 4,958,490 tokens in 9,686 chunks, fanned out across 20 GPUs and merged. Vocabulary coverage of the calibration corpus is 99.5%, so almost no embedding row was quantized without evidence.

Corpus composition: agentic tool traces 24.7%, code 17.8%, reasoning 14.8%, multilingual 13.8%, long context 11.9%, vocabulary sweep 9.9%, structured data 3.9%, graphics 3.0%. The vocabulary sweep is regenerated per tokenizer, since a sweep built for another model covers a different vocabulary and calibrates nothing here.

Reproducing our numbers

Everything needed to check this card is public:

• The BF16 file in this repo is the reference every measurement is taken against.
• The calibration corpus is AtomicChat/calib-corpora, build ornith-1.5-9b.
• Ornith-1.5-9B-GGUF-metrics holds the importance matrix, every raw KL divergence log including the losing ablations, and the reference logits themselves, so you can measure your own build against the same starting point instead of taking our word for it. To measure a build of your own:

llama-perplexity -m your-quant.gguf -f eval_neutral.txt \
  --kl-divergence-base base-neutral.kld --kl-divergence -c 4096 -ngl 99

The corpus and the context have to match ours or the numbers are not comparable.

About the model

Ornith-1.5 is trained with a loop that generates its own tasks, builds its own scaffolds, and rewards all three stages together rather than optimizing a policy against a fixed human written harness. For the method and the benchmark results, see the upstream model card and the Ornith blog. Those benchmark numbers are the authors' own and we have not independently reproduced them.