Qwen3.5-0.8B-Chat-MLX

Qwen3.5-0.8B Chat — MLX (Apple Silicon)

Text-only extraction of Qwen3.5-0.8B, quantized with MLX for on-device chat on Apple Silicon. Three bit widths are published so the quality/size tradeoff can be made with measured numbers rather than assumed.

INT8 is the recommended default. INT4 is kept for size-constrained use but is no longer recommended for anything that parses the reply as JSON — see Measured quality.

Model

Norm weights, A_log, dt_bias and the causal conv1d kernels are not quantized; they stay float. Quantized tensors are stored as MLX's weight / scales / biases triple with fp16 scales and biases.

Files

Each variant is self-contained: one directory is everything needed to run it. The vision tower and the multi-token-prediction draft head (mtp.*) are not included — no runtime here reads them. Dropping the draft head alone took 14 MB off the INT4 download compared with the previous revision.

Measured quality

Every number below is measured, not estimated. Method: the exporter's exact quantize/dequantize step is reproduced on the bf16 checkpoint — same kernel, same bit width, same group size — and the result is compared against the unmodified bf16 model on identical inputs. 32 768 teacher-forced tokens per corpus. A control run with quantization disabled reproduced bf16 exactly (0.000 % perplexity change, 100 % agreement, 0 KL), so the deltas below are the quantization and nothing else.

Perplexity is relative to the bf16 checkpoint; lower is better and 0 % is the target. Top-1 agreement is how often the quantized model's most likely next token is the same as bf16's.

Structured replies

24 tasks that ask for a single JSON object and are scored the way a strict parser would score them: greedy decoding, no repair pass, no leniency for markdown fences or prose. bf16 passes 24 of 24.

The INT4 failures are not near-misses. Five of the six are unbalanced braces — replies such as {"action_items": [{"owner": "Chen", "task": "..."]} that drop the object's closing brace, which no parser recovers from. The sixth invents a different key name. This is why INT4 is no longer recommended for any consumer that parses the reply: it degrades exactly where a downstream pipeline is least able to cope. INT5's single failure is the same kind of dropped closing brace.

Raising only the embedding to 8 bits while keeping the body at 4 was measured and does not rescue INT4: +15.4 % perplexity, 23 of 24 JSON. If structured output matters, use INT5 or INT8 rather than a mixed INT4.

Usage

Swift (speech-swift)

import Qwen3Chat

let model = try await Qwen35MLXChat.fromPretrained(quantization: .int8)
let response = try model.generate(
    messages: [ChatMessage(role: .user, content: "Hello!")],
    sampling: ChatSamplingConfig(temperature: 0.3, maxTokens: 100)
)

INT5 and INT8 need a runtime that reads the bit width from config.json. Older speech-swift versions build every QuantizedLinear and PreQuantizedEmbedding with bits = 4 hardcoded, which fits the INT4 file only; they will not read an INT5 or INT8 file correctly. Use a speech-swift version that takes quantization_bits and quantization_group_size from config.json, or stay on INT4.

Python

import json
import mlx.core as mx
from huggingface_hub import snapshot_download

path = snapshot_download("aufklarer/Qwen3.5-0.8B-Chat-MLX",
                         allow_patterns=["int8/*"])

cfg = json.load(open(f"{path}/int8/config.json"))
weights = mx.load(f"{path}/int8/model.safetensors")

# Quantized tensors are stored as a weight/scales/biases triple. Always take the
# bit width and group size from config.json rather than assuming them.
up = mx.dequantize(
    weights["layers.0.mlp.up_proj.weight"],
    weights["layers.0.mlp.up_proj.scales"],
    weights["layers.0.mlp.up_proj.biases"],
    group_size=cfg["quantization_group_size"],
    bits=cfg["quantization_bits"],
)
print(cfg["quantization"], up.shape)   # int8 (3584, 1024)

The layout is the custom key set the Swift runtime loads (layers.N.linear_attn.* for DeltaNet, layers.N.self_attn.* for the attention layers), not the mlx-lm naming, so mlx_lm.load will not read it directly.

CLI

# One variant only — do not pull all three
hf download aufklarer/Qwen3.5-0.8B-Chat-MLX --include "int8/*" \
    --local-dir ./qwen35-0.8b-int8

A note on earlier downloads

These files replace an earlier export in which the model's final RMSNorm weight was written without the +1 offset that Qwen3.5 stores its norm weights with. That mis-scaled the hidden state feeding the LM head and cost about +19 % perplexity on its own, on top of any quantization error. If you downloaded this repository before this revision, re-download it; the numbers in this card apply only to the current files.

Source

Converted from Qwen/Qwen3.5-0.8B (Apache-2.0). The vision tower and the multi-token-prediction head are dropped; the text model is quantized with mx.quantize at group size 64, the zero-centered RMSNorm weights are folded to 1 + w, and the DeltaNet conv1d kernels are transposed to MLX's channels-last layout.

Links

• speech-swift — Apple SDK
• Docs — install and CLI docs
• soniqo.audio — website
• blog — blog