Downloads · 30 days
79
5% of all-time downloads
Sahuagin/Ornith-1.0-35B-GGUF-router-intact
Ornith-1.0-35B-GGUF-router-intact is a text generation model from Sahuagin. Use it when you need the model to write or continue text. The card lists the license as mit.
Sub-Q4 GGUF quantizations of DeepReinforce's Ornith-1.0-35B that the official GGUF repo does not ship, built with asymmetric expert quantization: the ~93% of parameters in the routed experts are crushed to
Downloads · 30 days
79
5% of all-time downloads
All-time downloads
1.6K
Public
Repo size
50.8 GB
Likes
0
Public
Click a slice to open those files.
.gguf50.6 GB · 100%
From the Hugging Face model README
Sub-Q4 GGUF quantizations of DeepReinforce's Ornith-1.0-35B that the official GGUF repo does not ship, built with asymmetric expert quantization: the ~93% of parameters in the routed experts are crushed to 2/3/4-bit while everything the model steers with is kept at high precision.
This is a re-quantization of DeepReinforce's Ornith-1.0-35B using the asymmetric-expert-quantization technique pioneered by DeepSeek (FP4-experts/FP8-rest in V3/V4) and popularized for GGUF by Salvatore Sanfilippo (antirez). The only contribution here is the specific application — an importance matrix calibrated on agentic coding data — and the benchmarks below. All credit for the model belongs to DeepReinforce, for the technique to DeepSeek and antirez, and for the tooling to Georgi Gerganov and the llama.cpp contributors (binaries built by Unsloth).
| file | experts | everything else | size | fits 24GB card |
|---|---|---|---|---|
ornith-35b-expq4_K.gguf | q4_K | q8_0 (router f32) | 20 GB | yes (32k ctx) |
ornith-35b-expq3_K.gguf | q3_K | q8_0 (router f32) | 16 GB | yes (64k+ ctx) |
ornith-35b-expq2_K.gguf | q2_K | q8_0 (router f32) | 13 GB | yes (~131k ctx) |
imatrix-ornith35b.dat | — | importance matrix used above | 184 MB | — |
Precisely: routed experts (ffn_{gate,up,down}_exps) take the low-bit type; the router
(ffn_gate_inp, kept f32), attention, shared expert (ffn_*_shexp), embeddings and output
head stay at q8_0. The intuition: experts store knowledge, the preserved tensors carry
control flow (routing, attention, format discipline) — so agentic/tool-use ability survives
far below the usual quantization floor.
| rung | code review (score / recall) | agentic tool-loop | notes |
|---|---|---|---|
| q4 | 0.954 on a 5-case public-style suite; 0.78 / 0.85 on the 34-case suite | 73% | ties gemma4:31b / gpt-oss:20b as a local reviewer |
| q3 | 0.929 | 77% | |
| q2 | 0.897 | 80% | agentic ability statistically identical to q4 |
Treat these as relative signals (private fixtures), not leaderboard numbers. The stable findings across the study: recall stays perfect on seeded defects at every rung; review precision degrades mildly with expert bits; agentic tool-use does not degrade at all down to 2-bit experts.
# importance matrix on a RAM-fitting quant (activation stats are robust to base precision)
llama-imatrix -m ornith-q5.gguf -f calibration.txt -o imatrix.dat --chunks 400
# asymmetric quant from the bf16 GGUF: experts low-bit, rest q8_0, router stays f32
llama-quantize --imatrix imatrix.dat --tensor-type exps=q2_K ornith-bf16.gguf out.gguf q8_0
The included imatrix was computed on an agentic Rust-coding corpus (public code; per-tensor activation statistics only — no text is recoverable from it). For a general-purpose re-quant, substitute a broad calibration set.
n_seq_max = 1 regardless of parallel settings: one loaded instance serves
one request at a time. For concurrency, run N instances (one per GPU) — every rung here
fits a 24 GB card.FROM <gguf> with RENDERER ornith / PARSER ornith, stop <|im_end|>,
temperature 0.6, and a num_ctx that matches your VRAM plan.llama-quantize --tensor-type, llama-imatrix) by Georgi Gerganov and contributors; binaries built by
Unsloth.License: MIT, inherited from the base model.