GLM-4.7-Flash-DFlash

GLM-4.7-Flash-DFlash

A complete, trained DFlash draft model (~1.1B parameters) that accelerates GLM-4.7-Flash inference via speculative / block-parallel (draft-then-verify) decoding.

What this is

GLM-4.7-Flash-DFlash is a full drafter model trained to predict blocks of future tokens for GLM-4.7-Flash. A DFlash-capable runtime uses it to draft several tokens at once and verify them against GLM-4.7-Flash in a single pass, accelerating generation losslessly — the target model's output distribution is unchanged.

Speculative decoding always runs two models together: a fast drafter (this model) and the target it accelerates (GLM-4.7-Flash). Both are complete models; pairing them is how the speedup is obtained. This repo holds the drafter — the full weights (model.safetensors, ~2.1 GB), config, and results below.

Specs

Full hyperparameters are in config.json.

Intended use

Load as the drafter in a DFlash / speculative-decoding serving stack alongside GLM-4.7-Flash. The runtime loads both models; this drafter proposes token blocks and GLM-4.7-Flash verifies them, yielding the throughput gains shown below at no cost to output quality.

Method note: decoupling draft width from verification depth

A DFlash-style drafter normally uses a single width k for both the draft pass and the number of proposals the target verifies. In this work those two knobs are decoupled into independent, fixed-shape controls:

• draft_width — the non-causal DFlash draft-query width. Because mask rows attend to one another, a wider draft pass shifts the acceptance profile at early positions.
• verify_k — how many of those proposals are actually sent to the target for verification. On a fine-grained MoE target like GLM-4.7-Flash, each verified token is expensive, so this is the real cost driver.

The technique runs the proposer at a wide draft_width, keeps only the first verify_k proposals, and verifies just that prefix — discarding the tail before it ever enters the target's KV / sequence state. Draft and verify graphs stay fixed-shape (CUDA-graph capturable), and target output semantics are preserved (the DeepSpec prediction_shifted row contract), so the accelerated path is numerically faithful to the target.

Why it helps. A wide draft raises the number of committed tokens at early positions (≈3.1 committed through position 4 at width-8, vs ≈2.1 at width-4), while a short verify avoids paying MoE verification cost on the low-probability tail. The two optima are genuinely different, so tying them together leaves throughput on the table.

Results — vs. vanilla GLM-4.7-Flash

Against plain GLM-4.7-Flash with no speculative decoding, this DFlash drafter delivers roughly 1.3× higher generation throughput (≈ +30%) — about 121 → 157 tokens/s aggregate on 4×A100-40GB — while leaving the target model's output distribution unchanged. The speedup holds across every domain and is substantially larger on some (see plot).

Mean per-request output throughput across the full SPEED-Bench qualitative suite (880 prompts, 11 domains). Grey = plain GLM-4.7-Flash (no speculative decoding); DFlash (orange / teal) is faster across every domain. MTP-1 shown for reference.

Against a stronger MTP-1 speculative-decoding baseline the margin is naturally smaller (best DFlash config ≈ 1.16×, and ≈ +3% over the matched coupled setting at flat p95 latency) — but against the no-spec vanilla path the gain is large and consistent, which is the comparison that matters when deciding whether to use speculative decoding at all.

The best DFlash configuration is wide draft (8) / short verify (3) — the product of the decoupling described above. The full draft-width × verify-depth sweep:

Draft-width × verify-depth matrix: acceptance and throughput across configurations. Wide-draft / short-verify (e.g. 8 / 3) sits at the throughput optimum. Gains are workload-dependent, which motivates workload-aware / request-level routing rather than one static setting.

Training

Trained on 4×A100-40GB. Draft config: block 16, 8-layer draft (hidden 2048), auxiliary hidden states from target layers [1, 12, 23, 33, 44].

License

Derived from GLM-4.7-Flash — usage is subject to the base model's license. Verify and comply with the upstream terms before use.