Qwen3.8-27B-ABLITERATED-mlx-q3km

Qwen3.8-27B-ABLITERATED-mlx-q3km

MLX conversion of Blackfrost-AI/Qwen3.8-27B-ABLITERATED-BF16, quantized with the mixed_3_4 recipe as an equivalent to GGUF Q3_K_M.

• 3.910 bits per weight, 13.4 GB across 3 safetensors shards
• Architecture qwen3_5 (Qwen3_5ForConditionalGeneration), multimodal, hybrid linear/full attention every 4th layer
• Requires mlx-vlm >= 0.6.13 (earlier versions have no qwen3_5 module)

Why mixed_3_4 and not --q-bits 3

MLX's mixed-bit predicates are a deliberate port of llama.cpp's _K_M scheme, not an approximation. Both give more bits to v_proj and down_proj in the first eighth of layers, the last eighth, and every third layer between, plus a high-bit lm_head. mixed_3_4 is 3-bit base with 4-bit on those sensitive tensors, which is exactly Q3_K_M.

The measured result confirms it: 3.910 bpw here, against 3.91 bpw for the official GGUF Q3_K_M (13.30 GB for ~27.2B params). A plain --q-bits 3 would land near 3.5 bpw and would not match.

Note that mlx-vlm never quantizes the vision encoder at any level, so the vision tower stays bf16.

Conversion

mlx_vlm.convert \
  --hf-path Blackfrost-AI/Qwen3.8-27B-ABLITERATED-BF16 \
  --mlx-path ./Qwen3.8-27B-ABLITERATED-mlx-q3km \
  --quantize --quant-predicate mixed_3_4

Usage

pip install -U mlx-vlm

# text
mlx_vlm.generate --model wfiedler/Qwen3.8-27B-ABLITERATED-mlx-q3km \
  --prompt "Explain unified memory in two sentences." --max-tokens 256

# vision
mlx_vlm.generate --model wfiedler/Qwen3.8-27B-ABLITERATED-mlx-q3km \
  --prompt "Read this receipt and list every line item with its price." \
  --image receipt.png --max-tokens 400

Quality vs the 5-bit conversion

Benchmarked against the q5 build (5.678 bpw, 18 GB) of the same base model, on images with exact ground truth, 12 tasks x 3 trials at temperature 0.7.

On reading images the two are indistinguishable. Every OCR-shaped task ties perfectly: all receipt fields, all chart values, all table cells, every rare token in the small print.

The one real gap is counting many objects. Given 20 scattered shapes in four colour/type groups, q5 answered correctly three times out of three; this model answered 17, 19 and 21, never correct. The difference is visible in the output: q5 enumerates shape by shape before totalling, while the 3-bit build emits a terse number. The low-bit quant lost the habit of decomposing before answering.

Neither model does reliable mental arithmetic without a scratchpad; both compute a four-item subtotal wrongly at temperature 0.7.

Measured on an Apple Silicon Mac with 64 GB unified memory, each model in its own process. (GenerationResult.peak_memory is a process-global high-water mark that does not reset between models, so comparing two in one process makes the second inherit the first's peak.)

Use this build for OCR, document and chart work — 46% faster and 6.6 GB lighter for no measurable loss. Use the 5-bit build when the task involves counting or enumerating things in a scene.

Provenance and caveats

This is an abliterated model: the base has had refusal behaviour removed by its authors. It will comply with requests a stock instruct model would decline. Evaluate it before using it anywhere user-facing.

Quantization is lossy. The benchmark above covers a narrow slice of behaviour and says nothing about long-context, tool-calling, or code quality at 3 bits. All credit for the model itself goes to Blackfrost-AI and the Qwen team.