qwen3.8-27b-mlx-dflash2

qwen3.8-27b-mlx-dflash2

DFlash2 block-diffusion speculative decoding for Qwen3.8-27B, running natively on Apple Silicon (MLX).

Clone-and-run: OpenAI-compatible server, one-shot setup, test suite, reproducible benchmarks. 27B dense mxfp4 target + 5-layer DFlash2 draft = ~+57% decode throughput over dense on an M3 Pro 48GB.

┌────────────┐   hidden states    ┌──────────────┐
│  Qwen3.8   │ ───(layers 5/19/ ─▶│ DFlash2 draft│──▶ 4-token block
│  27B mxfp4 │     33/47/61)      │ 5L, parallel │    (one pass)
│  (target)  │◀───────────────────│  selector    │
└────────────┘   verify 5 tok     └──────────────┘
                  1 forward

Results (M3 Pro 48GB, cool silicon, greedy)

*Token-exact with dense when the prefill chunk size matches. bf16 chunked-GEMM reduction order shifts logits slightly; different chunk shapes can flip near-ties. Deterministic per config, semantically equivalent. See bench/LOSSLESS.md.

KV-8bit is free (same tok/s, half the memory). KV-4bit costs 2.3x decode. Sustained load throttles an M3 Pro to ~14 t/s after ~10 min (GPU downclock, not software).

Quick start (fresh clone → serving)

bash scripts/setup.sh          # venv + deps + models (~18GB) + text prep
bash scripts/serve.sh          # OpenAI API on :8006
python scripts/smoke_test.py   # verify: health, math, stream, think-cap

Or piecewise:

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
bash scripts/download_models.sh
python dflash2_mlx_server/server.py \
  --main-dir models/qwen3.8-text --draft-dir models/dflash2 \
  --port 8006 --block-size 4 --think-cap 12000

Test: bash scripts/run_tests.sh (unit + API smoke). Benchmark: python bench/test_dflash2.py 4.

Hardware: any Apple Silicon with ≥36GB unified memory (27B mxfp4 + draft + KV). Tested on M3 Pro 48GB.

Repo layout

The two integration patches (vs upstream z-lab/dflash)

1. _get_layers nesting — mlx_lm.load() on this checkpoint returns Model.language_model.model.layers (3 levels). Upstream hooks stop at 2. Applied in dflash2_mlx_server/server.py + bench/bench_common.py.

2. Chat template, not raw completion — the target emits <|im_end|> as token 1 on bare prompts (instant EOS). Always go through apply_chat_template(..., add_generation_prompt=True).

Hard thinking cap (12k default)

Qwen3.8 thinks by default. Unbounded <think> on a shared route burns the whole max_tokens budget before answering. The server enforces a hard cap (--think-cap, default 12000):

• Phase 1 generates with max_tokens = min(user_budget, THINK_CAP) and watches for the </think> token id.
• If the cap hits without closure, it force-injects </think>\n\n into the context and re-prefills, so the model still produces a full answer in the remaining budget (verified: cap=300 test produced a complete modular- arithmetic proof, 94% accept, think_capped: true in response stats).
• Normal closes skip phase 2 entirely — zero overhead.

Responses expose dflash2.think_capped so callers can tell a capped run from a natural one.

API

POST /v1/chat/completions (stream + non-stream), GET /v1/models, GET /health. Every response carries a dflash2 stats block (accept_rate, tok_per_s, think_capped) — use it to verify spec decode is engaged. Single-flight generation (MLX is single-stream): concurrent requests queue behind a lock rather than being rejected.

Production wiring (LiteLLM)

# litellm_config.yaml
model_list:
  - model_name: qwen3.8-27b-dflash2
    litellm_params:
      model: openai/qwen3.8-27b-dflash2
      api_base: http://<m3-tailscale-ip>:8006/v1
      api_key: "not-needed"
      request_timeout: 900
      extra_body:
        chat_template_kwargs:
          enable_thinking: false

Persistent macOS service (launchd, KeepAlive, auto-restart): deploy/install.sh.

Credits

• DFlash2 architecture + training: z-lab, inco.ai blog
• Target: mlx-community/Qwen3.8-27B-mxfp4
• Draft: z-lab/Qwen3.8-27B-DFlash2
• This repo: integration patches, server, benchmarks on M3 Pro.
• GitHub: https://github.com/fattchris/qwen3.8-27b-mlx-dflash2 · HF: https://huggingface.co/cfontes/qwen3.8-27b-mlx-dflash2

License: MIT (vendored z-lab code Apache-2.0).