Downloads · 30 days
0
ricardoholanda/Qwen3.8-Flash-Next-M5-Hermes-mlx-serve
Qwen3.8-Flash-Next-M5-Hermes-mlx-serve is a machine learning model from ricardoholanda. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for mlx. The card lists the license as mit.
<p align="center" <img src="hermes-qwen-flash-icon.png" width="180" alt="Hermes and Qwen Flash launcher icon" </p
Downloads · 30 days
0
Access
Public
Updated Aug 30, 2026
Repo size
1.8 MB
Likes
0
Public
Click a slice to open those files.
.png1.8 MB · 89%
From the Hugging Face model README
This is the reproducibility and integration release for running ddalcu/Qwen3.8-Flash-Next-MLX-Serve-4bit with mlx-serve and Hermes Agent on an Apple M5 Max MacBook Pro.
It does not duplicate the 98 GB model pack. The weights remain in the upstream model repository and retain the Qwen Community License. This companion repository contains the tested mlx-serve patch, benchmark results, the shared Hermes launcher, and the Hermes × Qwen Flash artwork.
This work builds on and responds to the original project and Apple M5/NAX
challenge created by David Dalcu (ddalcu):
Credit for the upstream Qwen Flash MLX conversion, the mlx-serve foundation, and the challenge definition belongs to David Dalcu. This companion release documents Ricardo Holanda's Apple M5 Max implementation, validation, benchmark, and Hermes integration work on top of that foundation.
| Area | Shipping behavior on the tested Qwen Flash fingerprint |
|---|---|
| Native MTP | Enabled by default |
| MTP depth | Adaptive from 1 to 3; never fixed at 3 |
| Depth 4 | Excluded from automatic selection after a measured workload cliff |
| Hermes reasoning | low by default in the dedicated Qwen profile |
| Dense M5 NAX lanes | Enabled through mlx-serve's hardware/geometry probes |
| Grouped expert NAX experiment | Correct and parity-tested, but off by default because it was slower |
| Hermes launch surfaces | Shared launcher for TUI and Desktop GUI |
The MTP controller chooses depth 1, 2, or 3 from live acceptance and measured
round cost. An explicit --mtp-depth remains an expert override. Hermes
reasoning effort and MTP depth are independent controls.
All figures are medians of three same-boot repetitions on an Apple M5 Max with 128 GB unified memory. The model is the exact q4/gs64 Flash-Next pack linked above.
| Workload | Adaptive MTP | Serial | Change |
|---|---|---|---|
| Code | 105.3 tok/s | 73.4 tok/s | +43.5% |
| Prose | 74.0 tok/s | 71.8 tok/s | +3.1% |
| 8.5K-token prompt | 68.8 tok/s | 64.6 tok/s | +6.5% |
| Cell | Before | After | Change |
|---|---|---|---|
| Decode | 91.8 tok/s | 99.4 tok/s | +8.3% |
| Prefill | 1861.4 tok/s | 1863.1 tok/s | +0.1% |
| Spec: predictable | 114.9 tok/s | 115.8 tok/s | +0.8% |
| Spec: novel | 72.0 tok/s | 73.8 tok/s | +2.5% |
The adaptive head averaged 2.86–3.00 speculative tokens per step in the final probe runs.
The new grouped q4/gs64 expert kernels passed FP32-dequantized parity at verify widths 2 through 8 and the full Qwen4 integration oracle. Performance still decides adoption: at width 2, 30-forward measurements were 22.189 ms grouped versus 18.553 ms stock, making the experiment 19.6% slower. It remains available for research with:
MLX_SERVE_MOE_VERIFY_NAX=1 ./zig-out/bin/mlx-serve \
--model "$HOME/.mlx-serve/models/ddalcu/Qwen3.8-Flash-Next-MLX-Serve-4bit" \
--serve
The production default is off. MLX_SERVE_MOE_VERIFY_NAX=0 is the explicit
kill switch.
The patch is based on mlx-serve commit
2a41d2e.
git clone --recurse-submodules https://github.com/ddalcu/mlx-serve.git
cd mlx-serve
git checkout 2a41d2e182136f6a8bdd4de9408a2b62dd36f102
git apply /path/to/mlx-serve-m5-hermes.patch
./scripts/fetch-zig.sh
./scripts/build-mlx.sh
env ZIG_GLOBAL_CACHE_DIR=.zig-global-cache \
./.zig-toolchain/zig build -Doptimize=ReleaseFast
Install the model pack with mlx-serve or download it from its upstream Hub repository. The tested launcher expects this location:
~/.mlx-serve/models/ddalcu/Qwen3.8-Flash-Next-MLX-Serve-4bit
The on-disk pack is approximately 98 GB, including the 32 GB memory-mapped n-gram table. Serving uses roughly 73–75 GB before KV-cache growth.
Copy hermes-mlx from this repository into the patched mlx-serve checkout as
scripts/hermes-mlx, then link it:
chmod +x scripts/hermes-mlx
mkdir -p "$HOME/.local/bin"
ln -sfn "$(pwd)/scripts/hermes-mlx" "$HOME/.local/bin/hermes-mlx"
Terminal UI:
hermes-mlx --tui
Hermes Desktop GUI:
hermes-mlx desktop
At the beginning of every CLI or Desktop GUI session, the launcher asks for:
Each launch receives a fresh directory under
~/Hermes Qwen Flash Sessions/, including workspace rules that prohibit
home-wide searches and require permission before accessing an outside path.
Hermes' file-writing tools are restricted to that session directory and its
dedicated profile. Broad shell searches rooted at the home, Documents,
Desktop, Downloads, or Library directories are blocked in the Qwen profile.
The launcher starts or reuses the local server at
http://127.0.0.1:11234, refreshes the dedicated profile under
~/.mlx-serve/hermes, applies the selected reasoning level, and leaves the
ordinary ~/.hermes profile untouched. Automatic post-turn memory/skill
review is disabled for this profile, so Qwen returns to idle after delivering
the answer; an explicit /refine still runs a review on demand. The MTP
controller remains adaptive from depth 1 to 3 regardless of the selected
reasoning effort.
On macOS, the launcher also keeps a synchronized runtime under
~/.mlx-serve/runtimes/qwen-flash-m5. This prevents dyld from being denied
access to MLX/llama libraries when the source checkout lives in the protected
Documents folder. The runtime refreshes automatically after rebuilding the
server or its staged libraries.
The shared menu-bar widget can now record labelled runs from either Hermes CLI or Hermes Desktop. Choose Start Stress-Test Recording…, run the workload, allow responses to finish, then choose Stop & Analyze. Reports are stored locally under:
~/.mlx-serve/analytics/hermes-qwen/
Each run contains raw two-second numeric samples (samples.jsonl), a stable
machine-readable summary (summary.json), a Markdown report, the MTP summary
lines emitted during that recording, and an advisory mtp-candidates.json.
The analyzer compares identically labelled prior runs and reports decode and
prefill throughput, latency, concurrency, queueing, utilization, memory, cache
tokens, cancellations, MTP acceptance, runtime self-disable events, and
per-context width cost.
The updated MTP log distinguishes the configured cap from the controller's
actual choices with avg_width and a width_hist of
width:rounds/accepted_tokens. Recommendations require trusted cost samples
and flag concurrent runs as unsuitable for per-width tuning. Candidates are
never applied automatically; validate them with controlled single-stream A/B
runs across code, prose, and long-context workloads first.
Install or rebuild the widget from this repository's included sources:
chmod +x scripts/build-hermes-qwen-metrics.sh
scripts/build-hermes-qwen-metrics.sh \
"$HOME/Applications/Hermes Qwen Metrics.app"
No prompt text, response text, filenames, or browser history is recorded.
A ready-made workload is included at
docs/market-web-four-agent-stress-test.md.
It launches four concurrent web-research agents, exercises official heavy-data
sites, and returns enough structured evidence to create a sizeable synthesis
prefill.
A second workload in Portuguese is included at
docs/brasil-ensino-medio-publico-4agentes-stress-test.md.
It launches four concurrent research and audit agents to compare Brazilian
public secondary-education outcomes across regions and states, with explicit
validity checks for SAEB, IDEB, ENEM, PISA, writing, and mathematics measures.
The recording format, statistical fields, comparison rules, and safe MTP
tuning workflow are documented in
docs/hermes-qwen-stress-analytics.md.
The release passed these live checks on the target M5 Max:
reasoning_effort: lowMachine-readable measurements and defaults are in
benchmark-results.json.
The automatic MTP policy is intentionally restricted to the exact measured
runtime fingerprint: qwen4_exp, 48 layers, hidden size 2560, 512 routed
experts, top-10 routing, intermediate size 640, affine q4/group-64 experts,
the checkpoint-native MTP head, and an available G17/M5 NAX surface. Other
geometries retain mlx-serve's generic opt-in policy.
ddalcu conversion; see the upstream
model card and its Qwen Community License.ddalcu/mlx-serve, MIT overall with incorporated components
under their respective notices.NOTICE and
LICENSE-APACHE-2.0 files.| File | Purpose |
|---|---|
README.md | This Hub model card and reproduction guide |
benchmark-results.json | Structured hardware, policy, and benchmark record |
mlx-serve-m5-hermes.patch | Implementation patch against mlx-serve 2a41d2e |
hermes-mlx | Shared Hermes TUI/Desktop launcher |
scripts/HermesQwenMetrics.swift | Menu-bar metrics widget and local statistical analyzer |
scripts/build-hermes-qwen-metrics.sh | Widget app-bundle builder |
docs/market-web-four-agent-stress-test.md | Reproducible four-agent, web-heavy large-prefill workload |
docs/brasil-ensino-medio-publico-4agentes-stress-test.md | Portuguese four-agent education-data workload |
docs/hermes-qwen-stress-analytics.md | Recording format, analysis metrics, and tuning safeguards |
hermes-qwen-flash-icon.png | Hermes × Qwen Flash artwork |
LICENSE, LICENSE-APACHE-2.0, NOTICE | License and upstream attribution |