Ornith-1.5-35B-A3B-oQ4e-mtp

Ornith-1.5-35B-A3B · MLX 4-bit (oQ4e) + MTP

MLX quantization of ornith-ai/Ornith-1.5-35B-A3B — a Mixture-of-Experts model with ~3B active parameters — in 4-bit (oQ4e) with Multi-Token Prediction (MTP, speculative depth 3), built to run on Apple Silicon via a local oMLX server.

Quantization details

• Base model: ornith-ai/Ornith-1.5-35B-A3B (qwen3_5_moe architecture, MoE with ~3B active parameters per token)
• Quantizer: oQ — oMLX v0.6.4, mixed-precision
• Vision encoder preserved — runs as a full VLM
• Report: oq_imatrix_report.json in this repo

Benchmarks on Apple Silicon (M1 Max 64 GB)

Measured on 2026-09-05 with the oMLX server on identical seed-fixed prompts (same-day runs): TurboQuant KV cache at 4 bits, DFlash2 speculative decoding (community draft jzinno/Ornith-1.5-35B-A3B-DFlash2), greedy decoding. Decode throughput, medians of two runs.

Decode speed (tokens/s) vs context length:

• Speculative decoding acceptance: 87–92% on generation tasks (~+70% tokens/s vs plain autoregressive decoding)
• Quality probes: needle-in-a-haystack recall 4/4 with zero false positives on all three variants (same fixed codes, ~23k context); factual, math and Italian-language checks all correct
• Agentic checks (headless runs of the pi agent, verified programmatically): 6/6 completed — bug-fix (3 planted bugs, unittest suite passes: 46/42/58 s for oQ4e/oQ6e/oQ8e) and multi-file analysis (5 CSVs into a verified summary.json: 33/42/52 s)

Takeaway: from 4-bit to 8-bit the throughput stays in the same band and retrieval recall is identical. On 64 GB machines the 6-bit variant is the sweet spot.

Sibling variants

• Ornith-1.5-35B-A3B-oQ4e-mtp — smallest footprint
• Ornith-1.5-35B-A3B-oQ6e-mtp — recommended on 64 GB machines
• Ornith-1.5-35B-A3B-oQ8e-mtp — highest fidelity

Credits

All credit for the base model goes to the Ornith team (ornith-ai):

• Base model: ornith-ai/Ornith-1.5-35B-A3B (MIT — see LICENSE), which extends Ornith-1.0 (built on Qwen3.5 and Gemma4) via an end-to-end self-improvement loop
• Blog: deep-reinforce.com/ornith.html · ornith.ai/ornith_1_5.html
• Quantization tooling: oMLX
• Speculative decoding draft: jzinno/Ornith-1.5-35B-A3B-DFlash2

This repo is only a quantization for local Apple Silicon inference. Same MIT terms apply.

Intended use

Local inference on Apple Silicon Macs (oMLX / mlx-lm). For base-model benchmarks and details, see the upstream card.