Qwen3.8-9B-Distill-MTPLX-Q6G64

Qwen3.8-9B-Distill-MTPLX-Q6G64

A local MTPLX forge of empero-ai/Qwen3.8-9B-Distill for native multi-token-prediction (MTP) speculative decoding on Apple Silicon (MLX). Quantized to 6-bit / group-size 64, with the native MTP draft head and the vision tower both preserved at full precision.

• Base model: empero-ai/Qwen3.8-9B-Distill (Qwen3-Next architecture, qwen3_5, 9B params, multimodal)
• Trunk quantization: 6-bit, group_size 64, affine mode (mlx mx.quantize) — ≈6.5 bits/weight effective (6-bit payload + 32/group_size bits of scale+bias overhead)
• MTP draft head: kept in native bf16 (not quantized) — speculative decoding is exact-rejection-sampling, so precision here affects speed/acceptance, never correctness
• Vision tower: preserved at bf16, 333 tensors (~912MB), grafted in from the source checkpoint
• Size on disk: 8.7GB (vs. 19.3GB source bf16)
• Context window: 262,144 tokens

Why this exists

mlx_lm's default conversion path drops the vision tower and MTP head for this architecture. This artifact restores both:

• The MTP head is calibrated fresh against the quantized trunk (Forge's own depth-sweep verification), so it's not just copy-pasted from the source — it's tuned for this trunk.
• The vision tower required a manual fix upstream of Forge: empero-ai/Qwen3.8-9B-Distill ships as a single unsharded model.safetensors with no model.safetensors.index.json, and Forge's vision-graft step (working around mtplx issue #263, where mlx_lm.convert silently drops vision weights) discovers tensors to copy only through that index. A synthesized index (mapping all 775 tensors to the single file) was added to the source cache directory so the graft could find and restore the 333 model.visual.* tensors. If you're forging this same source repo yourself and get "source declares vision_config but the forged artifact resolves no vision tower," this is why — sharded/indexed sources don't hit it.

Quantization recipe

{
  "body_bits": 6,
  "body_group_size": 64,
  "body_mode": "affine",
  "mtp_policy": "keep_bf16"
}

Speed (Apple M5 Pro, mtplx 2.9.1)

Forge's own depth-sweep verification (long-code-uncapped suite), sustained profile:

For comparison, sibling artifacts of the same source model, same hardware:

The 35B-A3B is a sparse mixture-of-experts model — only ~3B parameters activate per token despite 35B total, which is why it's the fastest of the four here at Forge's own calibration prompt despite being the largest on disk. It's also the only one of the three with its MTP head quantized to 4-bit rather than kept in bf16.

Decode speed vs. generation length

The table above uses Forge's fixed calibration prompt. Real workloads have varied completion lengths, and this model and the 27B behave very differently as length grows. Bucketed from the 164-problem HumanEval run below (excluding truncated-at-cap rows):

This model holds flat (even ticks up slightly) as completions get longer. The 27B trends down as completions get longer, and individual long completions dip as low as 22.8-23.8 tok/s (e.g. HumanEval/77 at 977 tokens: 22.8 tok/s). It isn't perfectly monotonic — some 1900+ token 27B completions still hit 36+ tok/s — so treat this as a real tendency with meaningful variance, not a hard curve.

Accuracy: HumanEval (pass@1, n=1, full 164 problems)

Real correctness eval — each completion is extracted and executed against HumanEval's actual test suite in an isolated subprocess (standard HumanEval methodology), not a similarity/vibes score.

Read this carefully before drawing conclusions: all three runs use different reasoning settings by design (this model at its most thorough effort tier, the 27B at its cheapest, and the 35B-A3B in its only mode — it has no effort dial, just auto/on/off for whether it thinks at all). This is not a controlled quantization- or size-only comparison. The 27B and 35B-A3B both have several times this model's parameters and still solved substantially more problems even at cheap/default settings, which says more about parameter count (and, for the MoE, total capacity despite sparse activation) than about any quantization scheme here. Don't cite this as "model X is Y% better than model Z at equal settings" — none of these are like-for-like tests.

† The 35B-A3B's auto reasoning mode turned out to think very extensively — 44% of its first-pass completions (72/164) hit the 2048-token cap without ever emitting an answer, compared to 9/164 for this model and 2/164 for the 27B. Those 72 were re-run with an 8000-token budget; 69 then passed, 1 genuinely failed, and 2 (HumanEval/132, HumanEval/145 — the same two problems that were hardest for the 27B too) hit even the 8000-token cap and are counted as failures. Combined compute time across both passes was ~107 minutes for the 35B-A3B, which isn't comparable to the single-pass timings for the other two models — this one needed a fundamentally different, more expensive procedure just to get an honest score, which is itself a relevant data point about this reasoning mode's verbosity on short, simple problems.

Note the completion lengths above ended up nearly identical (627 vs. 583 tok/problem) despite the very different reasoning-effort settings — so the speed gap between these two models is not mainly about one of them "thinking more." The real driver is throughput itself: this model averaged ~75.6 tok/s and the 27B ~30.2 tok/s over this same run — a ~2.5x gap, wider than the 1.6x gap on Forge's own fixed calibration prompt (see speed table above). The likely explanation is that the 27B's MTP draft head, calibrated against one long homogeneous coding prompt, accepts fewer speculative tokens on HumanEval's short, varied snippets than it does in-distribution; the 9B's held up better on the same variety.

Practical takeaway (measured directly on this same 164-task run, not extrapolated): this model at xhigh finished the full run in 22.7 minutes versus the 27B at low taking 52.8 minutes — about 2.3x faster wall-clock for a real batch of coding tasks, in exchange for roughly 5x the failure rate (18.9% vs. 3.7% of problems failed). Depending on how close a workload is to the 27B's own calibration style, expect the realistic speed multiplier to land somewhere in the 1.6x-2.5x range. Whether that trade is worth it depends entirely on how much your workload tolerates wrong answers versus wants iteration speed.

Methodology notes:

• Single sample per problem (pass@1 reduces to a plain pass rate at n=1), greedy-ish sampling via each model's own tuned sampler defaults (not forced temperature=0).
• A handful of problems (9 for the 9B, 2 for the 27B) initially hit the 2048-token completion cap mid-reasoning before emitting an answer; those were re-run with an 8000-token budget and the corrected pass/fail outcome is what's reported above. Uncorrected (2048-token-cap) raw numbers were 78.05% and 95.73% respectively.
• Full per-task results available on request; this card reports the aggregate.

Accuracy: real coding-agent benchmark (SWE-bench-style, 10 instances)

HumanEval above tests isolated single-function completion with no tools — the one dimension where these models are known to be closest (see the note in the next section). To probe real multi-step, multi-file agentic coding instead, we built a small SWE-bench-style harness: 10 real, verified GitHub issues (3 from pallets/flask, 7 from pytest-dev/pytest, drawn from SWE-bench Lite), each with a real repo clone reset to the exact pre-fix commit, a real Python environment, and real test execution — FAIL_TO_PASS tests must go from failing to passing, and all PASS_TO_PASS tests must keep passing (no regressions), exactly like official SWE-bench grading. Every instance's harness was independently validated end-to-end against the real upstream fix before any model was tested.

Tool quality dominates model quality here. A first attempt using a single generic bash tool (no dedicated file-edit tool) scored 0/10 for all three models — the models kept fumbling heredocs and sed to edit files instead of editing precisely. Switching to pi, a proper multi-tool coding agent (read/edit/write/bash) pointed at the same local MTPLX server, immediately unlocked real, differentiated results on the exact same instances. This matches an independent public finding for the related Qwen3.6-27B: bash-only tool scaffolding measured ~28% pass@1 on SWE-bench Pro, versus ~51% with a dedicated str_replace-style edit tool.

This ordering (27B > 35B-A3B > 9B) matches the qualitative ranking from official published benchmarks (Artificial Analysis Intelligence Index, SWE-bench Verified, Terminal-Bench) — a meaningful, believable signal, unlike the near-tie HumanEval produced.

Two clean patterns across the 10 instances:

• 3 instances (2 Flask, 1 pytest) were solved by all three models — genuinely tractable bugs regardless of model size.
• 5 instances were solved by none of the three models, and every non-timeout failure among them was the same failure mode: the target FAIL_TO_PASS test(s) started passing, but a previously-passing PASS_TO_PASS test broke. This looks like a property of those specific instances (a fix with wider blast radius than the test suite's FAIL_TO_PASS set captures) rather than a model-specific weakness.

Caveats: 10 instances is a small sample (not the full 300-instance SWE-bench Lite) — treat this as a directional signal, not a precise resolve-rate estimate. Each model used its own natural reasoning setting (not equalized) for the same reason described in the HumanEval section above. Pi's own model-name display is stale (always shows the default-configured model regardless of what's actually loaded); the true active model was verified independently via the server's /health endpoint before and during every run, not from Pi's self-report.

Usage

mtplx quickstart \
  --model pooyakn73/Qwen3.8-9B-Distill-MTPLX-Q6G64 \
  --host 127.0.0.1 --port 8000 \
  --reasoning-effort xhigh

OpenAI-compatible endpoint at http://127.0.0.1:8000/v1, model id qwen3.8-9b-distill-mtplx-q6g64. Depth-2 speculative decoding is applied automatically per this artifact's mtplx_runtime.json contract.

License

Apache 2.0, inherited from the base model empero-ai/Qwen3.8-9B-Distill, itself a distillation of Qwen/Qwen3.5-9B.