Qwen3.6-27B-Alis-MLX-Dynamic

Qwen3.6-27B-Alis-MLX-Dynamic

Apple Silicon (MLX) mixed-precision quantizations of Qwen/Qwen3.6-27B — a 27B dense hybrid (3:1 Gated DeltaNet : full attention, qwen3_5). One repo, one card — each quantization lives on its own branch.

* single-stream decode on an M3 Ultra, short prompt; longer contexts decode slower. † measured against its own ThinkingCap bf16 reference (different base than the other rows).

Everything below is measured, not projected — KL / top-1 flip on a fixed 4096-token EN/code/ZH/KO slice (teacher-forced vs the bf16 base), strided perplexity (ctx 2048 / stride 1024), and single-stream decode on an M3 Ultra.

Which build?

• 24 GB+ Mac → main. 14 GiB weights + 15.8 GB measured peak at short context; int8 KV gives ~48K context in 1.5 GiB of cache.
• 16 GB Mac → temporarily withdrawn. The 2.9bpw ThinkingCap tier passed teacher-forced KL/PPL checks but free-running generation on hard long-CoT problems showed early stopping and repetition loops (its calibration mix contained no long thinking traces — teacher-forced metrics cannot catch this failure mode). CoT-heavy recalibration fixed generation form (MMLU-300 77.0%) but GPQA-class reasoning stays collapsed at 3bpw (30.3%) — a precision floor. No 16 GB reasoning tier is feasible in this family; for reasoning at minimum size use the ThinkingCap 4.6bpw quant (15.47 GB, GPQA 83.3).

Multimodal (default)

main carries the full multimodal stack — 4.6bpw text weights + the vision tower at 8-bit (verified with EN/KO OCR tests). Plain mlx-lm loads main fine for text-only use (vision weights are skipped); the text branch saves 0.62 GB if you never need images.

pip install mlx-vlm
python -m mlx_vlm generate --model avlp12/Qwen3.6-27B-Alis-MLX-Dynamic \
  --image photo.jpg --prompt "이미지의 텍스트를 읽어줘." --max-tokens 300

Usage

from mlx_lm import load, generate

model, tokenizer = load("avlp12/Qwen3.6-27B-Alis-MLX-Dynamic")                      # main

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "뮤텍스와 세마포어의 차이를 설명해줘."}],
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=512))

CLI / server:

mlx_lm.generate --model avlp12/Qwen3.6-27B-Alis-MLX-Dynamic --prompt "..." -m 512
hf download avlp12/Qwen3.6-27B-Alis-MLX-Dynamic --local-dir qwen36-27b
mlx_lm.server --model ./qwen36-27b --port 8080

Long context: quantize the KV cache in the generate path (--kv-bits 8 --kv-group-size 64) — measured to be quality-free (below).

Recipes (verified from each build's config.json)

Per-tensor bits assigned by marginal utility from a 2–6-bit + nvfp4/mxfp4 weight-space sensitivity scan of all 498 text tensors, then validated end-to-end against uniform baselines (the scan alone is not trustworthy — see the nvfp4 finding below).

The golden spot for vanilla Qwen3.6-27B (survey of 102 MLX quants)

We surveyed every vanilla Qwen3.6-27B MLX quant on the Hub (102 repos; finetunes, abliterations and GGUF excluded) and measured the serious ≤16 GB contenders on our fixed KL ruler (4096-token EN/code/ZH/KO, teacher-forced vs bf16):

Nothing at or below this size class quantizes vanilla Qwen3.6-27B better — every lower-KL build we found is strictly larger (mlx-community 5-bit 19.4 GB, OptiQ-4bit 20.0 GB, Intel AutoRound int4 19.0 GB, …). This build is the quality-per-byte knee of the curve for the vanilla model. (For higher absolute quality at this size, use the ThinkingCap sibling: GPQA 83.3 vs this build's 72.7 — token-efficient bases quantize far better. This repo is the golden spot for the vanilla weights specifically.)

The nvfp4 finding. The scan's weight-space rel_error consistently rates nvfp4 (g16) above affine 4-bit (g64) at equal cost on this model — but end-to-end KL says the opposite: pure-nvfp4 4.5bpw scored 0.450 vs uniform affine-4bit's 0.383. Qwen's activation outliers appear to favor affine's per-group bias term. main is therefore affine-only; trust weight-space scans for ranking within a format, not across formats. Note the finding is architecture-dependent: on the 35B-A3B MoE sibling the reverse was measured — its nvfp4-mix beats an affine-only rebuild (0.292 vs 0.309).

DWQ. All tiers went through layerwise DWQ (KL distillation vs the bf16 teacher; 260-sample EN/KO/ZH/code chat-template mix, seq 512, validation-gated rounds — see alis-dwq). At ≥4bpw the gates rejected every round (the quantized model is already too close to the teacher for lr ≥2e-6 to help), so main carries its conversion-time weights. At 2.9bpw DWQ is decisive: the 16gb tier's valid KL dropped 0.436 → 0.298 (−32%).

Quality

KL(bf16‖quant) / top-1 flip on the fixed 4096-token slice, per domain:

Strided perplexity (ctx 2048, stride 1024; wikitext / code / Korean):

† different bf16 reference (ThinkingCap); PPL is comparable across rows, KL is not.

Generative benchmark (GPQA-Diamond, temp 1.0, our harness)

Long-CoT hard reasoning is far more quantization-sensitive than teacher-forced metrics suggest — measure generatively before trusting any low-bit build:

At ~3bpw generation form is fixable with CoT-heavy DWQ calibration (no more early stops/loops; MMLU-300 77.0%) but GPQA-class reasoning collapses — a precision floor, not a calibration artifact.

KV cache — measured recommendations

KV quantization drift is 2–3 orders of magnitude below weight-quantization drift (same-model logit KL, 2048-token chunked forward on the 16gb build):

Qwen3.6-27B is a 3:1 hybrid — only 16 of 64 layers carry KV at all; the other 48 are Gated DeltaNet with a fixed-size recurrent state. KV is already ~4× cheaper than a dense transformer of this depth before any quantization.

Notes & provenance

• main carries the full multimodal stack (mlx-vlm); text is the mlx-lm text-only artifact; mtp-bf16 carries the MTP head as a bf16 sidecar.
• Calibration/eval data: tulu-3 SFT chat (EN), Korean chat, code/ZH mix — chat-template rendered.
• Built on an M3 Ultra 512 GB · mlx-lm 0.31.3 · mlx 0.31.2 · 2026-07.
• Sibling repo: Qwen3.6-35B-A3B-Alis-MLX-Dynamic (MoE, 102–111 tok/s decode).

Credits

• Qwen team — the Qwen3.6-27B base model and its hybrid DeltaNet architecture.
• bottlecapai — ThinkingCap-Qwen3.6-27B, the token-efficiency finetune behind the 16gb-thinkingcap tier.
• Apple MLX team — mlx / mlx-lm, including the DWQ trainer this pipeline builds on.
• Unsloth — their NVFP4 releases motivated the diverse-chat calibration mix and served as quality reference points.
• AllenAI — tulu-3 SFT mixture used in the calibration set.
• Quantization recipe, DWQ retune and evals: Alis (avlp12) with the alis-dwq pipeline; built with assistance from Claude (Anthropic).