Ornith-1.5-35B-A3B-Q4_K_M-LocalMoe-M4Max

Deployment evidence update (2026-09-28): This card documents historical experiments, not a current minimum-speed or general agent-reliability guarantee. Later actual-workspace Pi tests exposed cold-prefill costs, client output-budget failures and answer-quality limitations. See the new reports and complete measured conditions. Historical evidence remains preserved.

LocalMoE: Ornith-1.5-35B-A3B on Apple Silicon

End-to-end engineering study and reproducible artifacts for serving Ornith-1.5-35B-A3B (35B total / 3B active sparse MoE) on an Apple M4 Max (64 GB unified memory). The original Q4_K_M configuration established the verified 262,144-token context result; the later Q6_K configuration adds a reasoning-on serving recipe and controlled short-workload comparison.

Previously documented on Q4_K_M (reasoning off). The Q4 model file was removed from disk under storage pressure and replaced with a SHA-256-verified Q6_K download; reasoning was enabled to recover the capability lost to the aggressive 4-bit quant. See RESULTS.md for the Q4→Q6 upgrade comparison. Raw Q4 records are retained in results/raw/.

Headline results

All numbers from identical-prompt controlled runs; raw JSON-lines in results/raw/. Methodology, negative results, and the head-to-head analysis in paper/main.pdf.

Why sparse wins for agents

A coding agent makes many model calls per task. The dense reasoning model pays two taxes on every call: thinking-chain tokens before any visible output, and speculative-decoding acceptance that collapses on open-ended prose. A 3B-active MoE pays neither — the result is a compounding 10–14× wall-clock advantage on real agent tasks even against a dense model accelerated by DFlash 2.

Negative results (pre-registered promotion gates)

Three speculation candidates were benchmarked and rejected: embedded NextN/MTP drafting, upstream no-spec builds, upstream ngram-mod. The MTP drafter that produced 809 tok/s on the dense target regressed most categories on the MoE. Full matrix: results/raw/speed_research_summary.json.

Repo layout

├── paper/            arXiv-style LaTeX paper (compiles with tectonic)
├── bench/
│   ├── verify_context.py        near-limit sentinel proof (260k)
│   ├── bench_raw_suite.py       streaming suite, cold + cached prefix
│   ├── run_pi_bench.sh          six-task pi agent suite
│   ├── bench_pi.sh              head-to-head pi harness (dense)
│   └── bench_pi_ornith.sh       head-to-head pi harness (MoE)
├── deploy/
│   ├── local-llm.command              production controller
│   ├── settings.json                  llama-server configuration
│   └── omlx-model-settings.example.json
└── results/
    ├── raw/                     all raw measurement records (JSONL/JSON)
    ├── head-to-head/            identical-prompt comparison artifacts
    ├── pi/                      original pi agent suite outputs
    └── *.log                    run logs

Quickstart (reproduce)

git clone https://github.com/minervacap2022/ornith-256k.git
cd ornith-256k
python3 reproducibility/verify_repro_bundle.py --remote

REPRODUCE.md pins the public Q4_K_M and Q6_K model revisions and SHA-256 values, llama.cpp revision, serving parameters, benchmark commands, and hardware envelope. It is the authoritative clean-machine procedure; the paper preserves the earlier Q4_K_M long-context experiment and the Q6_K addendum is documented in the manifest and raw Q4-versus-Q6 record.

Original-machine controller

The original M4 Max uses one Desktop controller shared with Qwen:

/Users/wilsonxu/Desktop/local-llm.command ornith start
/Users/wilsonxu/Desktop/local-llm.command ornith status
/Users/wilsonxu/Desktop/local-llm.command ornith test
/Users/wilsonxu/Desktop/local-llm.command ornith stop

This convenience controller is not required for reproduction on another host.

Verification gates (all passed)

1. /v1/models + /slots expose 262,144-token context ✓
2. Real 260,013-prompt-token request completes with exact sentinel recovery ✓
3. Streaming suite ×5 categories ×3 runs recorded ✓
4. Six pi agent tasks with independently verified filesystem effects ✓
5. Speculation candidates promoted only on broad win + separate context proof — none qualified ✗ (documented)

Related work

• chengyixu/qwen38-dflash2-bench — the dense LocalFlash baseline this study compares against
• ornith-ai/Ornith-1.5-35B-A3B-GGUF — model weights
• Raw records also mirrored at ChengyiX/ornith-256k-bench

Citation

@misc{klikteam2026localmoe,
  title={Small Active Expert, Big Context Window},
  author={Xu, Chengyi and {{KLIK team}}},
  year={2026},
  url={https://github.com/chengyixu/ornith-256k}
}

License

MIT — see LICENSE. Model weights subject to their upstream license.

<!-- minervacap-pre-hiklik-promotion -->

Discover Klik: https://pre.hiklik.ai/

<!-- /minervacap-pre-hiklik-promotion -->