Qwen3.8-35B-A3B-Distill-oQ8-fp16-mtp

Qwen3.8‑35B‑A3B‑Distill · oQ8‑fp16‑mtp

The fastest local model we have measured that passes our agent gate. This is a Novaeon.Studio oMLX build of empero-ai/Qwen3.8-35B-A3B-Distill: frontier‑distilled reasoning (Qwen3.8 teachers) in a sparse 35B / ~3B‑active MoE, quantized to near‑uniform 8‑bit with an fp16 router, and shipped with a native multi‑token‑prediction head that actually speeds up decoding.

It is the model every Novaeon agent runs on, 24/7, on an Apple M5 Max (128 GB). Every number on this page was measured by us on that machine, on these exact weights.

---

Contents

1. How it compares
2. Speed in detail
3. Agentic and tool use
4. What it is good at — and what not
5. Quickstart
6. Optimal oMLX settings
7. Hardware and memory
8. How we test
9. Model details
10. FAQ
11. Changelog · Attribution and license

---

How it compares

We don't choose our agent model from leaderboards. Every candidate runs through the same internal agent gate on the same Mac, and speed is measured head‑to‑head in the same session. This build has kept its seat against every challenger so far.

_{"—" = not run on that suite. Time for all 43 golden cases: this build 45–90 s · Gemma 153 s · APEX 178 s · Flash‑Next 269 s · dense 27B 367 s. Measured Sep–Nov 2026.}

Read it honestly. Gemma‑4‑26B scores two points higher on our single‑turn suite, and the dense 27B models are slightly more careful on the mitigation set — but they are 1.3× to 7× slower, and none of them beats this build on multi‑step agent work. Nemotron decodes ~11 % faster but fails the gate: it dropped parallel tool calls, looped run_python into the step limit and made ~45 % more tool calls in agent loops. For an always‑on agent worker on Apple Silicon, this build is the best speed/quality trade‑off we have measured.

---

Speed in detail

_{3 Nov 2026, oMLX 0.7.3, MTP on, thinking off, no other load. Five repeat runs on 30 Oct gave 114–134 tok/s short and 92–107 tok/s at 53k; ~216 tok/s aggregate with 4 parallel requests; cold load ≈ 1–6 s from a warm disk cache.}

Why it is fast: only ~3B of 35B parameters are active per token, the hybrid attention keeps a growing KV cache on only ~10 of 40 layers, and the native MTP head drafts tokens that the model accepts often enough to lift short‑context decode by ~40 % over MTP‑off.

---

Agentic and tool use

• Agentic suite A — 9/9, thinking on and off: multi‑step tasks in a sandbox with real tools (fix a bug, write code to a spec, analyse CSVs and logs, recover from tool errors, never delete files).
• Agentic suite B — 13/13, thinking on and off, 39 tool calls, no step‑limit loops.
• Tool‑use probes — 6/6: tool selection, parallel calls, sequential chains, argument fidelity, abstaining when no tool fits, no hallucinated tools.
• IFEval (ours, this build, thinking off): 84.86 avg, official verifier, 541 prompts / 834 instructions.

---

What it is good at — and what not

Use it for

• the default worker behind tool‑using agents, crons and chat ops — fast enough to sit in the hot path;
• multi‑step reasoning, coding, log/CSV analysis and structured JSON output;
• long documents (100k+ tokens) and screenshots/images in the same model;
• swarm lanes: several parallel requests share the GPU well (~216 tok/s aggregate at 4).

Know its limits

• Without thinking it makes arithmetic slips on multi‑step date/number puzzles (golden reason_arith, reason_cron_schedule). Turn thinking on for those, or route them to a dense model.
• Like every model we tested, it fails one hard 14‑step multi‑hop escalation task.
• It is a reasoning model: with thinking on, every answer starts with a <think> block. Strip it for end users.
• Our suites are internal agent tests, not public leaderboards. We publish the categories and every rival's score so you can judge the comparison.

---

Quickstart

# oMLX (recommended, Apple Silicon) — OpenAI-compatible server
omlx serve NovaeonStudio/Qwen3.8-35B-A3B-Distill-oQ8-fp16-mtp --port 8000

# Tool calling through the OpenAI-compatible endpoint
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="local")

tools = [{"type": "function", "function": {
    "name": "get_weather",
    "description": "Current weather for a city",
    "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}]

r = client.chat.completions.create(
    model="NovaeonStudio/Qwen3.8-35B-A3B-Distill-oQ8-fp16-mtp",
    messages=[{"role": "user", "content": "Weather in Bucharest and in Berlin?"}],
    tools=tools, temperature=0.6, top_p=0.95,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},  # fast, no <think>
)
print(r.choices[0].message.tool_calls)   # two parallel calls

# mlx-vlm
pip install -U mlx-vlm
python -m mlx_vlm.generate --model NovaeonStudio/Qwen3.8-35B-A3B-Distill-oQ8-fp16-mtp \
  --max-tokens 4096 --temperature 0.6 --top-p 0.95 \
  --prompt "A snail climbs 3 m by day, slips 2 m by night, in a 10 m well. How many days to escape?"

Recommended sampling: temperature 0.6 · top_p 0.95 · top_k 20. Thinking on for hard reasoning, off (enable_thinking: false) for agent loops and latency‑critical calls — that is how we run it.

---

Optimal oMLX settings

We swept the tuning levers on this build. Because the native MTP head genuinely accelerates decode under oQ8, the best profile is MTP on, with ANE prefill and TurboQuant KV off.

<details><summary>Drop‑in <code>~/.omlx/model_settings.json</code> entry</summary>

{
  "max_context_window": 262144,
  "mtp_enabled": true,
  "mtp_num_draft_tokens": 6,
  "a8_prefill_enabled": true,
  "a8_prefill_min_tokens": 128,
  "qwen35_ane_prefill_enabled": false,
  "turboquant_kv_enabled": false,
  "specprefill_enabled": false,
  "dflash_enabled": false,
  "moe_expert_offload_enabled": false
}

</details>

On the MTP head: empero ships the multi‑token‑prediction head natively in the checkpoint (model.safetensors.index.json maps the mtp.* tensors). oMLX recognises it (mtp_compatible: true) and it accelerates decode under oQ8. This is unusual: on other 35B‑A3B builds we tested, a "preserved" MTP head halved decode speed because drafts were almost never accepted.

---

Hardware and memory

_{Rows other than 128 GB are our estimate from the measured footprint, not tested machines.}

Memory stays flat with context: only ~10 of 40 layers carry a growing KV cache. Our 130k needle test peaked at ~47 GB.

---

How we test

• Hardware: Apple M5 Max, 128 GB unified memory, macOS, oMLX 0.7.x. The model under test runs alone; rivals are measured in the same session, after unloading the other.
• Golden (43): single‑turn agent cases — tool selection, parallel calls, JSON enforcement, reasoning with and without thinking, instruction following. A swap must not regress any tool‑category case.
• Mitigations (17): known failure modes — enforced JSON, reasoning with tools, swarm regressions.
• Agentic A (9) / B (13): multi‑step tasks in a sandbox with real file, Python and shell tools; a task only passes if the final answer is correct and no destructive or invented tool call happened.
• Speed: streaming decode and prefill at ~300, 41k and 124k tokens, thinking off, temperature 0.
• All runs at temperature 0. Scores given as a range (e.g. 39–40) are repeat runs or settings variants on the same weights.

---

Model details

Base‑model benchmarks (reported by the base authors, zero‑shot, vs the Qwen3.6‑35B‑A3B base it distills onto):

---

FAQ

Is it uncensored? No. This is the regular distill; we did not abliterate it. Uncensored variants we tested (Heretic, APEX Abliterated) scored lower on our agent gate.

Why 8‑bit and not 4‑bit? In our quant sweep of a sibling build, 2‑bit collapsed (13/43) and 4‑/6‑bit were not faster in practice, because with ~3B active parameters decode is already cheap. 8‑bit with an fp16 router keeps routing and tool reliability closest to the original at no real speed cost — if you have the memory.

Thinking on or off? Off for agent loops and anything latency‑critical (that is how the speed numbers are measured). On for hard math, dates and planning — it fixes most of the no‑think arithmetic slips.

Does it run outside oMLX? Yes, the weights load in mlx‑vlm. The MTP speed‑up and the tuned settings above are oMLX features.

---

Changelog

• 2026‑11‑03 — Card rebuilt: head‑to‑head comparison against 7 local models, new speed measurements on oMLX 0.7.3, updated serving profile (A8 prefill on, TurboQuant off), new Novaeon branding.
• 2026‑09 — First release: oQ8‑fp16 build with working MTP head, 6/6 tool‑use probe, IFEval 84.86.

---

Attribution and license

• Base model: empero-ai/Qwen3.8-35B-A3B-Distill — Apache‑2.0, by Empero. A distillation of Qwen3.8 frontier teachers into the Qwen3.6‑35B‑A3B architecture.
• This quantization, serving profile and benchmarks: Novaeon.Studio, 2026. Apache‑2.0, same as the base.

Independent community quantization, not endorsed by Empero. All credit for the model's capabilities belongs to the base‑model and Qwen authors. The header background is an AI‑generated image.

@misc{novaeon2026qwen38distilloq8,
  title  = {Qwen3.8-35B-A3B-Distill oQ8-fp16-mtp: an oMLX build for Apple Silicon},
  author = {Novaeon.Studio},
  year   = {2026},
  note   = {Quantization of empero-ai/Qwen3.8-35B-A3B-Distill},
  url    = {https://huggingface.co/NovaeonStudio/Qwen3.8-35B-A3B-Distill-oQ8-fp16-mtp}
}

_{novæon — Digital Business Architecture + AI · <a href="https://novaeon.studio">novaeon.studio</a>}