Downloads · 30 days
423
26% of all-time downloads
mchen04/Ornith-1.5-35B-A3B-REAP192-mxfp4-MLX
Ornith-1.5-35B-A3B-REAP192-mxfp4-MLX is a text generation model from mchen04. Use it when you need the model to write or continue text. It is set up for mlx. The card lists the license as mit.
A compressed derivative of ornith-ai/Ornith-1.5-35B-A3B built to run fast, swap-free, single-user inference on a base M4 Mac mini with 24 GB of unified memory — a machine on which the plain 4-bit conversion (19.5 GB)…
Downloads · 30 days
423
26% of all-time downloads
All-time downloads
1.6K
Public
Parameters
26.6B
14.2 GB on disk
Likes
3
Public
Click a slice to open those files.
.safetensors14.2 GB · 100%
How the weights are stored.
U3226.6B · 100%
From the Hugging Face model README
A compressed derivative of ornith-ai/Ornith-1.5-35B-A3B built to run fast, swap-free, single-user inference on a base M4 Mac mini with 24 GB of unified memory — a machine on which the plain 4-bit conversion (19.5 GB) does not run at all (hard Metal out-of-memory against the ~18 GB working-set cap).
This repo is the recommended quality-preserving build: 49.8 tok/s decode with task quality measurably above an uncompressed-expert 3-bit reference. A max-speed variant reaching 51.2 tok/s (at lower quality) is documented below and reproducible with the included scripts.
| Decode | Prefill (8k ctx) | Warm TTFT | Task quality* | NLL | Weights | |
|---|---|---|---|---|---|---|
| This build (recommended) | 49.8 tok/s | 451 tok/s | 0.31 s | 0.9814 | 0.8950 | 14.2 GB |
| Max-speed variant (recipe below) | 51.2 tok/s | 448 tok/s | 0.31 s | 0.9314 | 0.9039 | 14.1 GB |
| Reference anchor (unpruned, 3-bit experts) | 36.7 tok/s | 410 tok/s | 0.43 s | 0.9543 | 0.8977 | 16.1 GB |
*Composite pass-rate over a 42-task executable suite (unit-tested coding, exact-answer math, JSON tool calls, schema-validated structured output, format compliance, stability checks) plus rotated held-out task generations; NLL measured on a fixed mixed code/prose corpus. All numbers measured on a base M4 Mac mini (24 GB, macOS 26.2) serving over an OpenAI-compatible HTTP endpoint, median of repeated runs, greedy decoding. An independent fresh-context audit reproduced the recommended line bit-exactly on quality/NLL and within 0.2 % on decode.
Source: ornith-ai/Ornith-1.5-35B-A3B-MLX @ revision 1713baf2e38984a3c557195caf167759144d1d7e
(bf16, 69.3 GB). The Ornith team's card notes Ornith-1.5 was developed on top of Qwen3.5
and Gemma4 lineages; see the original model card
for training details and benchmark results of the uncompressed model.
Transformations, in order (each reproducible with the scripts in this repo):
num_experts_per_tok: 7 in config.json): −12.5 % active
bytes per token for +0.55 % NLL. k=6 was measured and rejected (three variants, all
fail quality gates).No fine-tuning, no weight editing beyond the above; the tokenizer and chat template are unchanged from the source.
Requires an Apple-silicon Mac (≈16 GB free memory; built for 24 GB machines) and mlx-lm ≥ 0.31:
pip install -U mlx-lm
# generate
mlx_lm.generate --model mchen04/Ornith-1.5-35B-A3B-REAP192-mxfp4-MLX \
--prompt "Write a Python function that merges overlapping intervals." --max-tokens 400
# OpenAI-compatible server (stock)
mlx_lm.server --model mchen04/Ornith-1.5-35B-A3B-REAP192-mxfp4-MLX --port 8080
Optional +8–10 % serving throughput: the stock server spends ~2.4 ms/token in its
batching/parsing layer. scripts/minimal_server.py is a drop-in single-user replacement
(same OpenAI-compatible API, adaptive prompt-prefix caching, reasoning/content split)
that serves this model at the bare-API decode rate:
python scripts/build_minimal_server.py # builds a patched runtime copy next to your mlx-lm
PYTHONPATH=./patched-minsrv python -m mlx_lm server --model <this-model> --port 8080
Quantize everything (including the shared expert) in mxfp4 instead of step 2 above —
recipe mxfp4 in scripts/quantize_variant.py, then the same pruning and k=7 config.
Costs ~5 task-quality points (0.9814 → 0.9314) for +1.4 tok/s; measured details in the
table. We recommend this build instead unless raw decode is all that matters. (A middle
build, 8-bit shared + 3-bit head, lands at 50.8 tok/s / 0.9564 — recipe mxfp4-sh8h3.)
This derivative changes efficiency characteristics only: no fine-tuning was performed and no safety-relevant components were intentionally altered. It inherits the behavior, alignment properties, and content policies (and their gaps) of Ornith-1.5-35B-A3B — refer to the original card. Compression can subtly shift refusal boundaries and factual reliability; the quality gate used here measured task competence, not safety behavior. Apply your own safeguards for user-facing deployments, and do not treat outputs as authoritative without verification.
scripts/ contains the exact tooling: quantize_variant.py (named recipes, pinned source
revision), reap_collect.py + reap_prune.py (saliency + pruning; prune-of-quantized is
exact because per-expert quantization groups are independent), build_minimal_server.py /
minimal_server.py (serving runtime), and DERIVATION.json (machine-readable provenance
for these exact weights). Deriving from scratch needs the 69.3 GB source checkpoint and
any Apple-silicon Mac with ~90 GB free disk (conversion is shard-streamed; 16 GB RAM
suffices).
@misc{ornith_1_5, title={{Ornith-1.5}: From Self-Scaffolding to Self-Improvement}, url={https://ornith.ai/ornith_1_5.html}, author={{Ornith Team}}, year={2026}}scripts/.License: MIT, inherited from the base model's declared license; see LICENSE.