Qwen3.8-27B-AEON-Ultimate-Multimodal-MLX-6bit

Qwen3.8-27B-AEON-Ultimate — Multimodal MLX 6-bit

A 6-bit MLX quantization of AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-BF16 (revision 8f76e82), AEON's abliterated release of Qwen/Qwen3.8-27B, for local inference on Apple Silicon.

The vision tower is fully preserved (333/333 tensors — AEON's release keeps it hash-identical to stock Qwen3.8), so image and video understanding work through mlx-vlm. The model's native MTP head is published separately as a drafter for lossless speculative decoding: VisualInference/Qwen3.8-27B-AEON-Ultimate-MLX-MTP-Drafter — pairing it typically speeds decode 1.4–1.9× at identical output quality.

Status (2026-09-10)

This build still works as documented below, and the vision path is unchanged. If you are choosing a Qwen3.8-27B for local agent use on a 48-64 GB Mac today, read this first.

What we run now, and why. After a measured comparison, our own lane moved to orcarouter/Qwen3.8-27B-Uncensored-MLX (6-bit) served by mlx-dspark with the incoai/Qwen3.8-27B-DFlash2 drafter. The full recipe, the harness results and the tooling are in qwen38-27b-mlx-agent-lane. The two builds differ in their abliteration, not in quantization, so the comparison is between bodies:

In plain terms: on agent behaviour the two are indistinguishable at one seed; in distribution this build sits about twelve times further from stock Qwen, which matches AEON's own published KL and their note that reasoning trails stock. If closeness to stock matters to you, prefer OrcaRouter. If you specifically want AEON's abliteration, this build remains the way to run it on MLX.

Run this build faster than the numbers below. The measurements in this card used mlx-vlm with the native MTP drafter. mlx-dspark (pip install mlx-dspark) with the DFlash2 drafter is lossless too and, on a 6-bit Qwen3.8-27B body at 24-27k tokens of context, decodes at 15-19 tok/s where the mlx-vlm MTP path measured 6-7 tok/s. The drafter was trained on stock Qwen3.8 and only affects speed, never output. Launcher settings and patches: qwen38-27b-mlx-agent-lane.

Two things to set for long agent sessions. Keep reasoning_effort: medium (at xhigh, Qwen3.8 issue #216 reports empty answers with finish_reason=stop). Add budget forcing: at a 4,096-token cap a few percent of open-ended turns end inside <think> with no answer; a small proxy that appends </think> and continues the answer removes every case we observed. Code: thinkbudget_proxy.py in the repo above.

Limits of these claims. One seed for the behaviour numbers, 8-30 prompts per test, one machine. Nothing here measures vision quality of the 6-bit tower.

Requirements

• Apple Silicon Mac with MLX support; pip install mlx-vlm (converted with mlx-vlm 0.6.13 / mlx 0.32.0).
• Unified memory: ~21 GB for weights, with an observed runtime peak of ~25 GiB at short context and ~30 GiB at 13k-token context. Comfortable on 48 GB+ machines, workable on 36 GB; not recommended below that.
• Because decode is memory-bandwidth-bound, tokens/sec scales roughly linearly with the chip's memory bandwidth (see measured numbers below).

Usage

# text / vision
python -m mlx_vlm generate \
  --model VisualInference/Qwen3.8-27B-AEON-Ultimate-Multimodal-MLX-6bit \
  --prompt "Describe this image." --image photo.jpg

# faster decode with the MTP drafter (lossless speculative decoding)
python -m mlx_vlm generate \
  --model VisualInference/Qwen3.8-27B-AEON-Ultimate-Multimodal-MLX-6bit \
  --draft-model VisualInference/Qwen3.8-27B-AEON-Ultimate-MLX-MTP-Drafter \
  --draft-kind mtp --draft-block-size 3 \
  --prompt "Write a short story about a lighthouse keeper."

Tips:

• Draft block size 3 is the all-round sweet spot; 4 edges ahead on code; ≥5 regresses.
• When serving (mlx_vlm.server), --prefill-step-size 512 cuts peak prefill memory by ~6 GB at no measured speed cost.
• Recommended sampling (from the Qwen3.8 card): thinking temp=1.0, top_p=0.95, top_k=20; non-thinking temp=0.7, top_p=0.8, presence_penalty=1.5.

Conversion recipe

python -m mlx_vlm convert \
  --hf-path AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-BF16 \
  -q --q-bits 6 --q-group-size 64 \
  --mlx-path Qwen3.8-27B-AEON-Ultimate-mlx-6bit

Affine mode, group size 64, no calibration (RTN is near-lossless at 6-bit). Language model and vision tower are both quantized at the global setting; mtp.* tensors are excluded by design — mlx-vlm loads the MTP drafter as a separate model (see the drafter repo for the split recipe).

Measured performance

Test machine: Mac mini M4 Pro, 48 GB unified memory (~273 GB/s). Higher-bandwidth chips (M-series Max/Ultra) scale decode proportionally — serial decode on this quantization measures at ~100% of the device's streaming-bandwidth roofline.

Prefill measured ~105–110 tok/s, flat with context length up to the 13k tested. Draft acceptance is workload-dependent: ~46% on open-ended prose, substantially higher on code and grounded QA.

Validation

• Text, vision, and long-context (13k) generation smoke-tested after conversion.
• Two-phase coding gate (spec research → implementation, executed against 28 hidden edge-case asserts across an SSE-parser task and a stack-VM task): 28/28.
• Speculative decoding verified lossless (drafter rejections fall back to the target model's own tokens; the sampling distribution is unchanged by construction).

Provenance & responsibility

Qwen/Qwen3.8-27B → AEON-7 SSM-conv1d repair + abliterix abliteration (BF16, vision and MTP untouched — see their card for methodology and KL evidence) → this repo (6-bit MLX quantization, nothing else changed).

This is an abliterated, refusal-removed model. As the upstream card puts it: the model does not decide whether to comply — you do. Outputs are the responsibility of the operator; use within the law of your jurisdiction. Apache-2.0, inherited from base.