GLM-4.1V-9B-Thinking-8bit

mlx-community/GLM-4.1V-9B-Thinking-8bit

zai-org/GLM-4.1V-9B-Thinking converted to MLX and quantized to 8-bit, for inference on Apple Silicon.

GLM-4.1V-9B-Thinking is a reasoning vision-language model that emits an explicit <think> block before its answer. The vision path is preserved in this conversion.

Existing MLX conversions of this model are 4-bit and 5-bit; this fills the 8-bit gap.

Quantization

Effective bits exceed the requested value because mlx-vlm quantizes only the language model and leaves the vision tower in bf16 by design — 181 vision tensors, none of them quantized. The vision encoder is a small share of the weights but disproportionately sensitive to quantization error.

language_model : 242 tensors quantized (8-bit, group size 64)
vision_tower   : 181 tensors, bf16    <- unquantized

Fidelity vs the original weights

Measured against the bf16 source, tensor by tensor, over all 242 quantized tensors (9,399,435,264 parameters). No prompts or sampling — a direct measurement of how much numerical information the quantization discarded, exact and hardware-independent.

Highest-error tensors — v_proj / k_proj and the LM head are the most sensitive:

rel_l2=0.00829  snr= 41.63 dB  language_model.lm_head
rel_l2=0.00815  snr= 41.77 dB  language_model.model.layers.15.self_attn.k_proj
rel_l2=0.00811  snr= 41.82 dB  language_model.model.layers.26.self_attn.v_proj
rel_l2=0.00776  snr= 42.20 dB  language_model.model.layers.32.self_attn.v_proj
rel_l2=0.00774  snr= 42.22 dB  language_model.model.layers.29.self_attn.v_proj

42.68 dB is the same figure measured for other 8-bit affine conversions in this series (e.g. MagenticBrain-8bit), which is what 8-bit at group size 64 should produce.

Throughput

M2 Pro / 32 GB, 64 generated tokens, greedy.

Numbers do not transfer across chips. Note this model emits a <think> block, so end-to-end latency per answer is higher than the token rate alone suggests.

Why fidelity is reported at the weight level

Other conversions in this series report perplexity ratio, top-1 agreement and KL divergence against the bf16 source. Those were run for this model and are deliberately not reported, because the measurement turned out to be invalid here — and that is worth stating rather than quietly omitting.

Teacher-forced on five plain-text passages, the numbers came out as perplexity ratio 3.99, top-1 agreement 0.568, KL 1.71 nats/token. Those would be alarming for an 8-bit quantization. But the same run also produced BLEU 83.92, chrF 87.67 and ROUGE-1 0.897 against bf16, and the sampled generations were byte-identical. Both cannot be true.

Breaking the aggregate down by passage explains it:

Aggregate perplexity is exp(sum_nll / sum_tokens), so the code block — 41% of the tokens, at a perplexity of 467 for the unquantized source — dominates everything. The bf16 reference spans 96x across these passages. Where the reference distribution is that flat, quantization noise flips the argmax easily, so top-1 agreement and KL understate fidelity rather than measuring it.

The weight-level comparison above has no such dependency on prompt choice, so it is what this card reports. For a model of this class a meaningful behavioural benchmark would need multimodal reasoning inputs, not plain prose.

What was not measured

No standard benchmarks: no MMMU, MathVista, MMBench, or any reasoning eval. No judged quality. The vision path was verified to load and run, not scored on a dataset. Evaluate on your own tasks before relying on this in production.

Usage

pip install mlx-vlm

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("mlx-community/GLM-4.1V-9B-Thinking-8bit")

prompt = apply_chat_template(
    processor, model.config,
    "What is shown in this image? Reason step by step.",
    num_images=1,
)
out = generate(model, processor, prompt, image=["image.png"], max_tokens=512)
print(out.text)

Give it room in max_tokens — the model spends tokens inside <think> before answering. Text-only works too: pass num_images=0 and omit image.

Note that stock mlx-lm loads the text path only; use mlx-vlm for image input.

Credits

All credit for the model belongs to Zhipu AI / Tsinghua KEG (Z.ai). This is a format conversion and quantization; no training or fine-tuning was performed. Licensed MIT, as the original. See the original card for intended use and limitations.