GLM-4.7-Flash-APEX-GGUF

GLM-4.7-Flash — APEX GGUF

[!IMPORTANT] Requires llama.cpp b10251 or newer. These files keep the MTP (nextn) draft head, which older builds do not know how to create, so they refuse to load the file:

error loading model: done_getting_tensors: wrong number of tensors; expected 868, got 862

That is a loader version problem, not a corrupt download — the file has all 868 tensors, but a pre-b10251 build only asks for 862 because it has no GLM-4.7-Flash MTP support (#24868, commit 57c0921). Update llama.cpp and it loads. Open an issue here if you need an MTP-stripped build for an older runtime.

MoE-aware, mixed-precision APEX quantization of zai-org/GLM-4.7-Flash — 31.2B total / ~3.6B active, and the most-downloaded model in the GLM family by a wide margin.

Three tiers, all built from the bf16 checkpoint (no intermediate requantization), all imatrix-guided, all with the MTP draft head preserved and quantized so speculative decoding works out of the box.

Architecture, and why it quantizes well

Despite the GLM name this is a DeepSeek-lineage design — llama.cpp converts it via Glm4MoeLiteModel(DeepseekV2Model) to the deepseek2 GGUF architecture.

Two properties make it an unusually good APEX target:

• Routed experts are 89% of all parameters (27.8B of 31.2B). Everything whose error compounds on every token — MLA attention, shared experts, embeddings — is only ~2.1B, so all of it stays at ~6.5–6.9 bpw for about 1.8GB while the entire remaining budget goes to experts.
• moe_intermediate_size is 1536, cleanly divisible by 256, so all three expert matrices (gate/up/down) can use the full k-quant and IQ menu. Models with a non-256 expert dimension cannot: their ffn_down_exps is restricted to block-32/64/128 types.

MLA keeps long context affordable. Instead of caching full K and V per head, it caches one 576-dim latent per token per layer:

Tiers

Allocation is by tensor role and layer band: edge layers keep more bits than the redundant middle, routed experts absorb the compression, and shared experts / MLA projections / embeddings stay high. The MTP draft head is pinned to Q4_K in every tier (see below).

On the PPL column: the imatrix is deliberately code-weighted (~70% multi-language code, 30% prose) and these numbers are measured on wikitext, so they are the pessimistic direction of that domain mismatch. The task benchmark below tracks quality much better — i-quality gives up 5.2% PPL but only 3.6 points of task score.

Agentic coding benchmark

Five hard from-scratch and fix-existing-code challenges driven through a real agentic loop (OpenCode), scored by partial credit over independently re-run pytest — never the agent's own self-report. MTP speculative decoding enabled (n_max=2). One run per tier.

This is a sweep, not a cutoff claim. Three sizes are shipped with their measured degradation so you can pick against your own budget — one model, one workload, one run per tier is not enough to assert a general "quantize no further than X bpw" threshold, and none is asserted here.

Read honestly:

• i-quality and i-compact are indistinguishable (both 53/59) despite a 24% difference in expert precision and 3.3GB of file size. Between 4.03 and 3.05 bpw the compression is effectively free on this workload.
• i-mini is genuinely weaker — 70.4%. It holds on the shorter challenges and degrades on the longest, most compositional one (expr_parser_advanced, 20/20 → 7/20). This is the one tier where PPL and task score agree that something real was lost. It is published because a 10.2GB file competes against other ~10GB models, not against the larger tiers here — but that comparison is not benchmarked, so judge it on its own numbers.
• bf16's 2-test-case lead is a single run on one challenge and is within plausible run-to-run variance for a stochastic agentic loop. Treat it as "bf16 is not clearly better than i-quality" rather than a measured 3.6-point gap.
• Wall clock mixes decode speed with how long each tier flailed on the hard challenge. The cleanest same-work comparison is expr_parser_advanced, identical 20/20: bf16 404s vs i-quality 157s (2.6×).

MTP speculative decoding

The nextn draft head is preserved and quantized (not stripped). llama-imatrix never exercises it — a plain forward pass reports the whole MTP layer as unused — so it receives no imatrix data, and is pinned to Q4_K rather than an IQ type, which would be fitted against statistics that do not exist for it.

Measured on i-mini:

Draft acceptance across tiers, from the benchmark runs:

Acceptance is ~59–65% for every quantized tier, from 4.03 down to 2.24 bpw experts — the speculative speedup survives aggressive quantization intact. Notably the bf16 file, whose draft head is also bf16, posts the lowest acceptance of the four: a low-precision drafter is not a compromise here, which matches independent results on other MoE checkpoints.

Enable it with:

llama-server -m GLM-4.7-Flash-APEX-i-quality.gguf \
  --spec-type draft-mtp --spec-draft-n-max 2 \
  --jinja -ngl 999 --ctx-size 32768

Plain decode speeds without MTP: i-quality 64.2 t/s, i-compact 71.4 t/s, i-mini 78.7 t/s.

Usage

# chat / agentic serving (add --spec-type draft-mtp for the MTP speedup)
llama-server -m GLM-4.7-Flash-APEX-i-quality.gguf --jinja -ngl 999 --ctx-size 32768

# one-shot
llama-cli -m GLM-4.7-Flash-APEX-i-quality.gguf --jinja -ngl 999 --ctx-size 8192 -st \
  -p "Write a Python LFU cache with O(1) get and put."

This is a reasoning model — it emits thinking before its answer, so allow a generous token budget or responses will be truncated mid-thought. For multi-turn agentic use, ZAI recommends Preserved Thinking mode; llama.cpp exposes --reasoning-preserve.

Requires llama.cpp b10251 or newer (see the note at the top): Glm4MoeLite conversion support plus GLM-4.7-Flash MTP support (upstream #24868).

Reproducing

• Converted from the bf16 checkpoint with convert_hf_to_gguf.py (no intermediate requantization).
• imatrix: llama-imatrix, ctx 512, over a ~70% multi-language code / 30% prose corpus, published at Myric/Laguna-S-2.1-imatrix-calibration-study (corpora/laguna_calib_codeweighted.txt). Expert coverage 100% except 9 expert stacks at 98.44% (63 of 64 experts observed).
• Per-tensor allocation emitted as a --tensor-type-file consumed by stock llama-quantize; no kernel or C++ changes.
• PPL: llama-perplexity on wikitext-2 test, ctx 512.

Attribution & licenses

• Base: zai-org/GLM-4.7-Flash (MIT)
• Engine: llama.cpp (MIT)
• APEX: localai-org/apex-quant (@mudler) (MIT)
• Calibration corpus: permissively-licensed sources (MIT / Apache-2.0 / BSD / Unlicense / public-domain prose)

Unofficial community quantization; not affiliated with or endorsed by ZAI.