Downloads · 30 days
35
15% of all-time downloads
xero0000/Vortigaunt-35B-mixed-q2k-imat
Vortigaunt-35B-mixed-q2k-imat is a text generation model from xero0000. Use it when you need the model to write or continue text. It is set up for gguf. The card lists the license as mit.
Black Mesa mixed-quant series · the deliberate reasoner.
Downloads · 30 days
35
15% of all-time downloads
All-time downloads
234
Public
Repo size
18.2 GB
Likes
0
Public
Click a slice to open those files.
.gguf18.2 GB · 100%
From the Hugging Face model README
Black Mesa mixed-quant series · the deliberate reasoner.
A CPU-offload-aware mixed-precision GGUF of Ornith-1.0-35B — deepreinforce-ai's
RL-tuned reasoning model on the qwen35moe architecture. 256K context at ~78 tok/s
decode on 18 GB of VRAM (vs ~61 tok/s for the uniform Q4_K_M), with perplexity within
~1% of the lossless Q8_0.
TL;DR: the strong-reasoner Ornith, running the full 256K window at small-context speed, in a 17 GB file.
Ornith-1.0-35B shares the hybrid qwen35moe architecture (gated attention + gated-delta-net
SSM, 256 experts, 8+1 active, ~3B active params/token) but is RL-tuned for reasoning. This
build applies the same mixed q2_K + imatrix quantization that the Qwen3.6 mixed quants use:
the expert layers that get offloaded to system RAM are dropped to Q2_K (roughly half the
bytes/token streamed from DDR4), while GPU-resident tensors stay at Q4_K. Decode is bound by
RAM bandwidth on offloaded layers, so cutting those bytes is what buys the speed.
qwen35moe).Q8_0 GGUF, requantized with llama-quantize (--allow-requantize).Q8_0 (~61K tokens / 119 chunks) so the
Q2_K rounding is guided by this model's activation statistics — not borrowed from a
different model.ffn_*_exps on blocks 13–26 → Q2_K (CPU-offloaded set, 42 tensors)Q4_K (GPU-resident) · output-class → Q6_KSee the Gordon (Qwen3.6 base) mixed quant card for the full explanation of the offload-byte-count rationale.
RTX 3060 Ti (8 GB) + RTX 3080 (10 GB), Ryzen 5950X, 46 GB DDR4-2733, ik_llama.cpp,
q4_0 KV, flash-attn on:
| Metric | This mixed q2_K | Ornith Q4_K_M | Ornith Q8_0 |
|---|---|---|---|
| Decode | ~78 tok/s @256K | ~61 tok/s @256K | ~32 tok/s @64K |
| Context | 262144 | 262144 | 65536 |
| VRAM | ~16.5 GB | ~15.4 GB | ~15.6 GB (+~26 GB RAM) |
| Tool-calls | clean Qwen tool_calls | clean | clean |
| Perplexity (holdout, ctx 512) | 2.3676 | — | 2.3434 |
Perplexity A/B vs the Q8_0 reference: +1.03% (near the ±0.06 confidence interval).
Ornith's RL-tuned weights are slightly more sensitive to the Q2_K layers than the base
Qwen3.6 (which is +0.59%), but the loss is still small for a 2.4× decode speedup and 4× context.
The -ot override is required — it pins the Q2_K layers (13–26) to CPU so the low-bit
tensors are exactly the offloaded ones. Do not add --n-cpu-moe with this build.
./llama-server \
-m ornith-1.0-35b-mixed-q2k.gguf \
--jinja --cache-type-k q4_0 --cache-type-v q4_0 --flash-attn on \
--ctx-size 262144 --parallel 1 --n-gpu-layers 99 \
-ot 'blk\.(1[3-9]|2[0-6])\.ffn_(up|gate|down)_exps\.weight=CPU' \
--tensor-split 44,56 --ubatch-size 256 --no-mmap --threads 8 --no-warmup \
--port 8000
Ornith is a reasoning model — it emits a chain of thought before answering. Keep reasoning
enabled for hard problems; it produces clean Qwen tool_calls for agent use.
Q2_K expert layers are the quality floor; this model loses a touch more than the
Qwen3.6 mixed quant (~1% PPL). For maximum fidelity use the Q8_0.Released under the base model's MIT license.