Qwen3.8-27B-CRACK-DWQ-4bit-MLX

Qwen3.8 27B CRACK — DWQ 4-bit MLX

This is a 16.20 GB, standard-runtime MLX-VLM derivative of dealignai/Qwen3.8-27B-MXFP8-CRACK.

• The language tower is affine 4-bit, group size 64, refined with 128 deterministic DWQ calibration records.
• All 417 vision tensors are value-identical to the source.
• The 31-tensor native MTP shard is byte-identical to the source and remains embedded in this repository.
• The source CRACK chat template is retained byte-for-byte.
• Loading and generation use official, unmodified MLX packages. No runtime fork or custom model code is required.

[!IMPORTANT] The source model has weight-level refusal ablation. It may comply with unsafe requests. You are responsible for safe and lawful use.

Usage

Tested on Apple Silicon with MLX 0.32.0, MLX-LM 0.31.3, and MLX-VLM 0.6.13. Because this repository contains the complete vision-language model, load it with MLX-VLM, not the text-only mlx_lm loader.

python -m pip install "mlx==0.32.0" "mlx-lm==0.31.3" "mlx-vlm==0.6.13"

mlx_vlm.generate \
  --model WaveCut/Qwen3.8-27B-CRACK-DWQ-4bit-MLX \
  --prompt "Explain why the sky is blue." \
  --max-tokens 256 \
  --temperature 1.0

mlx_vlm.generate \
  --model WaveCut/Qwen3.8-27B-CRACK-DWQ-4bit-MLX \
  --image ./image.jpg \
  --prompt "Describe this image." \
  --max-tokens 256 \
  --temperature 1.0

The inherited chat template enables xhigh reasoning by default. It also supports medium and low; pass enable_thinking=False or use the equivalent runtime option to disable thinking. The source card recommends temperature=1.0, top_p=0.95, top_k=20, with EOS token IDs 248046 and 248044.

What was quantized

DWQ optimized only the language-tower affine scales and biases against sparse teacher logits from the MXFP8 source. The 128 unique calibration records contain 48 tool-calling, 32 SWE-agent, 32 multilingual, and 16 code examples. The public source datasets were NousResearch/hermes-function-calling-v1, nebius/SWE-agent-trajectories, CohereLabs/aya_dataset, and openai/openai_humaneval. Images were not used for language-tower DWQ; vision was protected by exact tensor preservation plus an official-runtime image smoke.

Evaluation

Agent/tool decision sample

We used a deterministic, stratified 96-row sample from NVIDIA When2Call and teacher-forced choice log-likelihood with the native Qwen tools template. All models saw the same selected rows.

Against source MXFP8, this model was uniquely correct on four rows and uniquely wrong on zero: +4.17 percentage points, paired bootstrap 95% interval [+1.04, +8.33] points, exact two-sided McNemar p=0.125. Against RTN, it was uniquely correct on two rows and uniquely wrong on zero: +2.08 points, paired bootstrap interval [0.00, +5.21], McNemar p=0.5. This is bounded evidence on a small agent/tool sample, not a claim of universal superiority.

Tool hallucination was unchanged at 5/16 eligible rows for all three models. Length-normalized scoring tied RTN at 38/96 and scored 38/96 versus 37/96 for source MXFP8.

Distillation proxy and runtime smokes

• Fixed held-out sparse-teacher loss: RTN 0.168036 → DWQ 0.093706 (44.24% lower).
• Official MLX-VLM text smoke: exact answer 4, 18.26 GB peak.
• Official MLX-VLM image smoke: exact answer Blue, 18.34 GB peak.
• The source CRACK template scored 42/96; replacing only the template with Frog Qwen Fixed Chat Templates v22.1 scored 40/96. The full Frog template is therefore not included.

These memory figures are short-context measurements on an Apple M2 Max with 64 GB unified memory. Longer context and KV cache increase memory use.

Native MTP status

The original MTP tensors are preserved in this repository; no separate companion model was published. Official MLX-VLM can split and load them:

python -m mlx_vlm.speculative.drafters.qwen3_5_mtp.split \
  --model WaveCut/Qwen3.8-27B-CRACK-DWQ-4bit-MLX \
  --output ./qwen38-crack-mtp

Preservation does not imply a speedup with this new q4 target. On one fixed 128-token greedy prompt with draft block size 2, speculative output matched the ordinary output exactly, but accepted 0/128 draft tokens and ran at 12.50 tok/s versus 21.60 tok/s without MTP. Keep MTP disabled by default unless your own prompt/runtime benchmark demonstrates positive acceptance and throughput.

License and attribution

Apache-2.0, following the source model. The base Qwen model is by Alibaba; the MXFP8 CRACK source and refusal-ablation work are by dealignai. This derivative changes language-tower quantization only and retains the source safety profile.