Ornith-1.0-397B-Featherweight

Ornith-1.0-397B Featherweight — a 397B agent in 119.5 GB

397B-parameter, 512-expert MoE compressed to 2.41 bits/weight, verified against its own FP8 original on a frozen 604-case behavioral suite: function-calling 0.91, multi-turn tool use 1.00, code 0.98, composite S = 0.9258 (vs FP8's 0.8693 under a bare protocol — read the paper before quoting the comparison).

The base model, deepreinforce-ai/Ornith-1.0-397B, is a state-of-the-art open agentic-coding model (per DeepReinforce: SWE-bench Verified 82.4, Terminal-Bench 2.1 ~78, competitive with 1.6 T and frontier closed models — their benchmarks). This repo makes that model runnable on consumer hardware and proves the behavior survives compression; the capability numbers are DeepReinforce's, the fidelity numbers are ours (paper has both).

<p align="center"> <img src="https://huggingface.co/deepreinforce-ai/Ornith-1.0-397B/resolve/main/assets/ornith_397b_eval.png" alt="Ornith-1.0-397B agentic-coding benchmarks (DeepReinforce)" width="70%"> </p>

_{Base-model capability benchmarks — DeepReinforce's published evaluation, reproduced for context (not re-run by us).}

How it was made, validated, and every dead-end (~100 hours of autonomous ratchet research): → PAPER.md · research report · harness + goldens + journal. This page is only what you need to host it.

Files

Quickstart

Just run the autotuner. It probes your GPUs/RAM/cores, picks the fastest engine you have installed, and computes offload/context/thread flags from the measured serving laws — then launches. One command, any machine:

./featherweight-autotune.sh            # add --dry-run to see the computed command first

That's the whole story for most people. The details below are only if you want to drive the engines yourself.

Engine tiers (what autotune picks for you)

Bare zero-install boot (no dependencies, portable-safe defaults — serves the OpenAI API on :8080; append your own -ngl/-ts/--n-cpu-moe or just use autotune):

chmod +x Ornith-1.0-397B-Featherweight-serve.llamafile && ./Ornith-1.0-397B-Featherweight-serve.llamafile

On NVIDIA, install llama.cpp — the llamafile's driver-only CUDA (tinyBLAS) is a compatibility fallback: ~7× slower than cuBLAS and occasionally fails to initialize. It exists so the file runs everywhere, not so it runs fast. A self-contained binary with in-process autotune (like the sibling gemma-4 llamafile) is on the roadmap — it needs that fork's bundled llama.cpp rebuilt against qwen3_5_moe.

Expected speed by hardware

Predictions follow the serving law (decode ≈ RAM bandwidth ÷ ~1.5 GB streamed/token; paper § "Run it yourself"). The ± ranges are honest priors, not measurements — post your rig's numbers via the 7-step smoke checklist (paper) to the results thread and this table tightens.

Recommended hardware

2×24 GB GPUs + ≥90 GB fast RAM + NVMe — the verified reference class (the box the whole research program ran on). Below that it still works: RAM size decides fit (≥120 GB comfortable; less streams from SSD), RAM bandwidth decides decode speed, GPUs mostly buy prefill plus a decode bonus from expert offload.

Operational notes (measured, not folklore)

• llamafile default port is 8080 (--port N to change); any flag you append overrides the baked defaults — last value wins.
• Single GPU: --n-cpu-moe 60, no -ts. Autotune OOM? FEATHERWEIGHT_GPU_LAYERS=8 ./featherweight-autotune.sh.
• Mac under 128 GB: raise the wired limit first — sudo sysctl iogpu.wired_limit_mb=<RAM_MB×0.94>.
• The model thinks by default (9K+ chars on trivial prompts if unbounded): keep --reasoning-budget 1024.
• Never enable speculative decoding on CPU-resident-expert rigs — measured −10% (ngram) to −34% (draft model); mechanism in the paper. Ornith's declared MTP head has no published weights in any repo.
• Off-llama.cpp servers (vLLM/SGLang): tool calls arrive as <function=…> XML in content — parse it or tool benchmarks read as zero. llama.cpp --jinja handles it natively. (Also: vLLM currently lacks SM120 FP8-MoE kernels — on workstation Blackwell use SGLang.)

Credits & thanks

This is a derivative work standing entirely on others' shoulders — please credit them too:

• The model: deepreinforce-ai/Ornith-1.0-397B and its FP8 release by DeepReinforce — MIT-licensed weights of a genuinely impressive 397B agentic MoE. None of this exists without them.
• The inference engines: ggml-org/llama.cpp and GGML (Georgi Gerganov & contributors) — the runtime everything is served on; Mozilla-Ocho/llamafile (Justine Tunney & contributors) for the zero-install path; sgl-project/sglang for FP8-MoE serving on Blackwell.
• The method: karpathy/autoresearch — the ratchet-loop pattern this whole program is a port of; and antirez/ds4 (Salvatore Sanfilippo), whose asymmetric-crush 2-bit MoE recipe this independently reproduces on a different architecture.
• The evaluation data: gorilla-llm BFCL (Berkeley Function-Calling Leaderboard, Apache-2.0) and evalplus HumanEval+ (MIT/Apache-2.0) — per-artifact licensing table in the dataset card.
• Infrastructure: Hugging Face for Inference Endpoints (and the quota support that made the FP8 anchor possible).

License

MIT (base model MIT). Full provenance, licensing table, eval harness, FP8 goldens, and the append-only research journal live in SEBK4C/molt-ornith-eval.