Ornith-1.0-35B-MTP-GGUF

Ornith-1.0-35B MTP GGUF (Q4_K_M / Q6_K / Q8_0)

Three MTP-grafted quantizations of deepreinforce-ai/Ornith-1.0-35B — each shipped with a measured agentic verdict, not a perplexity chart. All three were benchmarked head-to-head on identical machine-scored task batteries (multi-turn agentic, long-horizon reasoning, code-with-hidden-tests; full harness + journals: SEBK4C/ornith-35b-agentic-eval).

TL;DR — which file to download

The finding: a quant-sensitivity ladder

On this RL-post-trained agentic model, quantization damage does not hit capabilities uniformly. Measured order of fragility:

1. Agentic persistence breaks first (Q4) — Q4_K_M aces every single-shot reasoning cell (58k-token needle chains, 17-hop code traces, 99-step recurrences) but abandons multi-turn tasks: 2-3 turns in, required output files never written, trajectory just ends. The RL-trained "diligence" is the most quant-fragile thing in the model — and no perplexity metric shows it.
2. Format precision gets noisy next (Q8-vs-Q6 trajectory level) — exact verbatim-output discipline (keep a required prefix, exact line counts) flips with seed at both Q6 and Q8; 3-seed study: Q6 5/6 vs Q8 3/6 on the discordant cells.
3. Reasoning is robust all the way down — exact-answer math/logic/retrieval held at every quant we tested.

Practical rule: don't requant an agentic RL model below Q6 without a multi-turn diligence eval. Single-shot benchmarks will lie to you.

Why is Q8 not better than Q6?

Across two independent 10-task hard-battery runs plus a 4-cell brutal battery: Q8_0 never won a single cell against Q6_K, ran 22% slower, and consumed +13-40% more thinking tokens per battery. A control run (official Q8_0, no MTP, different llama.cpp build) reproduced the Q8-MTP failures cell-for-cell — excluding serving machinery. Hypothesis (not proven): Q8's rollouts deliberate longer without converging better on this always-reasoning model; part of the observed gap is seed-level trajectory divergence (quantified in the dataset repo's seed-variance journals). What is proven: there is no upside to paying Q8's cost.

MTP / speculative decoding notes

• All three files carry the 20-tensor MTP head (blk.40.* incl nextn.*). Serve with --spec-type draft-mtp --spec-draft-n-max 3 on a llama.cpp with MTP support (merged 2026-05, PR #22673).
• MTP survives requantization: draft acceptance measured identical (47%) on Q4 and Q6 with matched prompts. The head is not the fragile part.
• MTP inverts under batch: measured crossover at ~2 concurrent streams (with MTP: 205/220/256/286 t/s agg at 1/2/4/8 streams; without: 171/~250/ 349/451; at 16: 321 vs 514). Enable MTP only for workloads that are mostly 1-2 concurrent; disable for fleets. (serving/*.args encodes this.)
• -np 1 is NOT required on current builds (stale early-notes claim; verified to np16+).

Serving (what we actually run)

See serving/ for complete flag sets. Summary (2×24GB GPUs):

• Sampling (model card): --temp 0.6 --top-p 0.95 --top-k 20.
• KV: q4_0 measured clean at 58k-token retrieval depth; the hybrid-attention architecture makes KV ~9.4GB per 1M tokens at q4_0 — context is cheap.
• --reasoning-budget 10240 recommended for work profiles: the model overthinks; the budget force-closes <think> so an answer always emits (converts silent token-cap deaths into honest attempts).
• Instant no-think mode per request: "chat_template_kwargs": {"enable_thinking": false} — ~2× cheaper on mechanical multi-step tool work, near-parity quality; keep thinking ON for combinatorial planning.

Fixed chat template (serving/ornith-35b-fixed.jinja)

The template embedded in upstream GGUFs hard-raises (HTTP 500 through llama-server) on three real-world client behaviors. This repo ships a fixed template, byte-identical to upstream on all happy paths (offline render-diff proven), lenient on:

1. developer role → rendered as a system block (OpenAI SDK / Codex clients send this; upstream raises Unexpected message role).
2. Mid-conversation system messages → rendered in place (upstream raises).
3. Tool-continuation histories with no plain user query → rendered (upstream raises No user query found in messages; the guard's last_query_index is dead code in the 35B template).

Serve with --chat-template-file ornith-35b-fixed.jinja --jinja --reasoning-format deepseek.

Known model-family bugs (all quants, reproducible)

• -2**2 evaluates to 4: the model consistently binds unary minus tighter than ** (8/8 attempts across Q4/Q6/Q8, with and without self-testing). Self-verification does not fix it — the model writes its tests from the same false belief. Harness rule: supply golden test vectors in prompts; never prose-only specs.
• Unaided full-context aggregation (~700 records) exceeds reliable capability at any quant/reasoning depth — but a shell workspace solves it in ~1.4k tokens via self-scaffolded grep|awk (the model was RL-trained to generate scaffolds; harness it shell-first).

Files

Ornith-Serve: one-file tuned serving (Linux x86_64 + NVIDIA)

Ornith-Serve-x86_64-cuda.run (29GB, self-extracting) bundles a patched llama.cpp server + CUDA runtime libs + the Q6_K-MTP weights + the fixed template + all three measured profiles behind one launcher:

./Ornith-Serve-x86_64-cuda.run          # extract once
./ornith-serve/ornith-serve.sh fastest  # or: fast | parallel

What the bundled build has that stock llama.cpp doesn't: MTP self-speculative decoding for this model, the fixed chat template, measured profiles, and an experimental --spec-max-active batch-occupancy gate for speculative decoding (default off; drafts only while ≤N streams generate — the one-server-all-modes design is in progress, see Honest limitations). Requires the NVIDIA driver; ~24GB VRAM (fastest), 2×24GB recommended for parallel. sha256: 6a30cfba93c715ebe82d5de7a3dd486f08e4399eee353fa9ef1d106e3cdb2b73

A true cross-OS llamafile is planned once llamafile's llama.cpp sync includes MTP (its current engine predates it).

Provenance & attribution

• Base model: deepreinforce-ai/Ornith-1.0-35B (MIT) — agentic-coding RL post-train of Qwen3.5-35B MoE. Official GGUFs ship without the MTP tensors.
• MTP graft method: skinnyctax/Ornith-1.0-35B-Q6_K-Frankenstein-MTP-GGUF — 20 head tensors appended, metadata patched (block_count+1, nextn_predict_layers=1). No weights retrained.
• Q6_K-MTP file: re-hosted from wang-yang/Ornith-1.0-35B-MTP-GGUF (MIT), unchanged.
• Q8_0-MTP: same graft procedure applied to the official Q8_0 GGUF.
• Q4_K_M-MTP: llama-quantize --allow-requantize from the Q8_0-MTP file (deterministic; requant noise is part of the published finding).

Honest limitations

• Benchmarks are single-seed unless marked; the 3-seed variance study covers the Q6-vs-Q8 discordant cells only. Cell counts are small (6-24 per battery); we publish direction + full journals, not significance claims.
• All numbers from one host (2×RTX 4090, llama.cpp-mtp build 4fc4ec5 era).
• --spec-max-active (in the Ornith-Serve build) gates drafting correctly and preserves solo speed, but spec-armed state currently carries a fixed per-iteration cost inside llama_decode that caps gated batch throughput (~223 vs 514 t/s @16 streams). Until that's profiled out, use the parallel profile (spec off) for fleets — the gate is for burst protection on small-slot profiles.
• The eval tasks are freshly authored (contamination-resistant) but ours; the full harness is published for scrutiny and reruns.