Ornith-1.5-35B-A3B-REAP-50-NVFP4A16

Ornith-1.5-35B-A3B REAP-50 NVFP4A16 (16 GB)

REAP expert-pruned (50%) + GPTQ-NVFP4A16 quantized build of ornith-ai/Ornith-1.5-35B-A3B, sized and served to run as a local coding model inside 16 GB of consumer VRAM — 12.47 GiB, RTX 5070 Ti (SM120), vLLM. Text-only: the base model's genuinely-trained vision tower and its MTP draft head are deliberately removed (documented trade-offs below), which is what makes the text-only architecture (Qwen3_5MoeForCausalLM) servable by stock-class vLLM after two confirmed upstream gaps were patched locally (both independently fixed upstream since, in vLLM #50210).

Methodology — how this and the sibling 16 GB builds are pruned, quantized, and evaluated (with the failure modes): https://github.com/t-timms/blackwell-16gb-moe

First quality signal — HumanEval+ 84.15% [77.8, 88.9], HumanEval 90.24% [84.7, 93.9], MBPP+ 89.15% [85.6, 91.9], greedy decoding, instruct framing, measured on exactly the released checkpoint. Agentic: SWE-bench Verified 22/50 = 44.0% resolved (81.5% resolved-of-completed), via mini-swe-agent's official bash-only scaffold at a 49K-token context ceiling, graded by the official harness with zero infra failures — measured on the same 50-instance slice as the prior release's 52.0% figure. The run is context-limited (15/50 hit the ceiling before submitting); see the SWE-bench section below before citing the headline number without that context.

Architecture

graph TD
    Base["ornith-ai/Ornith-1.5-35B-A3B<br/>qwen3_5_moe - 256 experts - MTP head - ~65 GB bf16 - MIT"]

    subgraph Build ["Build pipeline - RTX 5070 Ti, SM120"]
        REAP["REAP expert prune 50%<br/>256 -> 128 experts + router-renorm fix (architecture-detected)"]
        MTP["Strip MTP head<br/>mtp_num_hidden_layers 1 -> 0 - reload-verified"]
        StripV["Strip vision tower<br/>333 visual tensors (genuinely trained - scope trade-off)"]
        Quant["GPTQ-NVFP4A16 quantize<br/>compressed-tensors - ~5.5 h - 12.47 GiB"]
    end

    subgraph HF ["Published on Hugging Face"]
        A16["REAP-50-NVFP4A16 - 12.47 GiB<br/>vLLM - the depth artifact"]
        GGUF["REAP-50-GGUF<br/>Q4_K_M / Q5_K_M / Q6_K / Q8_0"]
        BF16["REAP-50-bf16<br/>pruned source for AWQ / EXL2 / MLX"]
    end

    subgraph Serve ["Serving & evaluation"]
        vLLM["vLLM - SM120 (3 upstream gaps patched locally)<br/>prefix cache 96.5% - 49K ctx"]
        Bench["HumanEval+ 84.2% - MBPP+ 89.2%<br/>SWE-bench Verified 44.0% (mini-swe-agent, same 50-instance slice)"]
    end

    Base --> REAP --> MTP --> StripV --> Quant --> A16
    StripV --> BF16
    BF16 -. convert .-> GGUF
    A16 --> vLLM --> Bench

Highlights

Why 50 percent

Forced by arithmetic on a 16 GB card, not a tuning choice:

Supporting evidence for the ratio: Half the Experts, All the Code pruned Qwen3.6-35B-A3B — this base model's size-class cousin — at 50% with no statistically detectable loss on its primary code benchmark.

Evaluation

HumanEval+ and MBPP+ via lm-eval-harness, greedy decoding, instruct framing, Wilson 95% confidence intervals, served through vLLM on the released checkpoint itself:

These are single greedy draws, and the intervals above describe sampling error only. do_sample: false with a fixed seed pins token selection but not vLLM's batch composition, and FP4 reductions are not order-invariant, so the score moves between byte-identical re-runs. Measured 2026-09-05 on a sibling release (KAT-Coder-V2.5-Dev REAP-50 W4A4 — same harness, same vLLM / SM120 NVFP4 stack): five draws spanned 1.85–4.27 pp depending on the task. That spread has not been measured for this checkpoint specifically, so treat each score above as one sample and read gaps of a couple of points as unresolved rather than real.

The full 706-problem suite ran in ~12 minutes through vLLM's backend. For tier context, our previous release (KAT-Coder-V2.5-Dev REAP-50 NVFP4A16, a different and code-specialized base) reports HumanEval+ ~90% and MBPP+ ~90% on its own card, on the same tasks: this build is a few points lower but in the same tier, consistent with more aggressive pruning (same 50%) applied to a general-purpose MoE base rather than a code-specialized one. No upstream HumanEval/MBPP numbers are published for Ornith-1.5-35B-A3B itself, so there is no published figure to compare against directly.

SWE-bench Verified — read before citing the 44.0% figure alone

Measured 2026-08-26 via mini-swe-agent's official bash-only scaffold (the prior release's protocol), graded by the official SWE-bench harness with zero infra or eval errors, on the released checkpoint itself:

The 50 instances are the same shuffled slice the prior release's 52.0% figure comes from — a same-instance comparison under the same scaffold, window, and step limit. Prior release (KAT-Coder-V2.5-Dev REAP-50 NVFP4A16, code-specialized base): 26/50 resolved, 26/32 = 81.25% resolved-of-completed. This build: 22/50, 22/27 = 81.5% resolved-of-completed. The headline gap is entirely submission rate (33 vs 27 instances reached a real completion attempt), not patch quality once submitted — resolved-of-completed is indistinguishable. Where the missing submissions went: 15 ContextWindowExceededError (both builds fail this way — the 16 GB KV budget is the binding constraint, not the model), 6 LimitsExceeded (the prior release had none at step_limit 65; instances this base finishes less turn-efficiently), 2 RepeatedFormatError. 18 instances were resolved by both builds, 8 only by the prior release, 4 only by this one. Consistent with the code-benchmark tier picture above: same tier, slightly behind the code-specialized prior release.

Config note. Sampling followed the checkpoint's own generation_config.json (temperature 1.0 / top_p 0.95 / top_k 20) — the vendor-documented default, adopted as the starting point and flagged as unvalidated in scripts/swebench/ornith_overrides_ladder.yaml; the prior project's precedent is that this class of choice must survive a full pilot before being trusted, and no sampling sweep has been run here. Window: max_model_len 49,152 with a measured 54,067-token total KV budget (engine-reported max concurrency 1.10x at that window; the ceiling varies 40–61K tokens with desktop VRAM contention, and the ladder harness falls back to smaller windows automatically). Read the headline as context-limited, not unconditional: 15 of 50 instances never submitted because they ran out of window. Raw artifacts (trajectories, preds.json, grading report) are preserved with the run; reproduction entry points: scripts/swebench/run_ladder_night.sh, scripts/swebench/grade_pilot.sh.

Quantization and pruning details

Usage

Requires vLLM with SM120 support. The configuration below is the one actually exercised end-to-end during evaluation (via vLLM's engine, greedy):

vllm serve Ttimms/Ornith-1.5-35B-A3B-REAP-50-NVFP4A16 \
  --dtype bfloat16 --trust-remote-code \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 8 \
  --max-cudagraph-capture-size 8 \
  --mamba-cache-mode align \
  --enable-prefix-caching \
  --mamba-block-size 528 --block-size 528

Notes, each earned the hard way:

• --max-cudagraph-capture-size below vLLM's default of 512 avoids a documented mamba-cache assertion for this architecture family; 8 is what the evaluated configuration used.
• --mamba-block-size 528 --block-size 528 alignment is required for correct hybrid-cache sizing on this checkpoint.
• Prefix caching works, but is ineffective for prompts shorter than the 528-token mamba block boundary (open upstream issue vllm-project/vllm #40696) — expect no prefix-cache benefit on short early turns in agentic workloads.
• The code benchmarks ran with max_model_len 2048; the SWE-bench pilot served the same released checkpoint at 49,152 with a measured 54,067-token total KV budget — the measured serving ceiling on this 16 GB card (it varies 40–61K tokens with desktop VRAM contention).
• Greedy (temperature=0) is what the HumanEval+/MBPP+ numbers were measured with. The SWE-bench pilot instead used the checkpoint's own generation_config.json sampling (temperature 1.0 / top_p 0.95 / top_k 20) — the vendor-documented default, adopted as the starting point, not a validated choice; see the config note in the SWE-bench section.

Known limitations

• No pruning-ablation baseline measured. The unpruned model does not fit this hardware; the accuracy cost of REAP itself (independent of quantization) is not isolated here. The comparison against the prior release above is tier context, not an ablation.
• Text-only by construction. The vision tower was trained and worked; this build cannot see images. Documented trade-off, reversible in principle by rebuilding from the unstripped variant.
• No MTP / speculative decoding. Forfeited by the MTP-strip decision above.
• Context-limited agentic results. The SWE-bench pilot served at a 49K window; 15/50 instances hit ContextWindowExceededError before submitting, so 44.0% is a floor under this hardware budget rather than an unconditional capability number. The ceiling itself moves (40–61K tokens total) with desktop VRAM contention on this card.
• These are self-reported numbers with published reproduction scripts, independently re-runnable from the eval suite in the companion repository; they are not leaderboard submissions.

Prior art and scope of claims

Verified against the Hugging Face Hub on 2026-08-25:

• Unpruned NVFP4 of this base exists officially: ornith-ai/Ornith-1.5-35B-A3B-NVFP4 (~21.8 GiB).
• REAP-pruned Ornith builds did not exist on the Hub as of that date.

What is distinct, and all that is claimed: a vLLM-servable Ornith that fits 16 GB with KV-cache room, with published HumanEval+/MBPP+ and SWE-bench Verified numbers, Wilson intervals, and the exact serving configuration — none of which the unpruned quant above publishes. The upstream-gap fixes required to serve text-only Qwen3.5-MoE checkpoints have landed upstream (vLLM #50210), so current vLLM builds serve this architecture without local patching.

License

MIT, inherited from the base model ornith-ai/Ornith-1.5-35B-A3B.

Citation

This checkpoint is derived from ornith-ai/Ornith-1.5-35B-A3B. If you use it, please cite the base model:

@misc{ornith15_2026,
  title={Ornith-1.5-35B-A3B},
  author={{Ornith AI}},
  year={2026},
  url={https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B}
}

And the pruning method:

@misc{reap2025,
  title={{REAP} the Experts: Why Pruning Prevails for One-Shot {MoE} compression},
  author={Lasby, Mike and Lazarevich, Ivan and Sinnadurai, Nish and Lie, Sean and Ioannou, Yani and Thangarasa, Vithursan},
  year={2025},
  eprint={2510.13999},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2510.13999}
}