gemma-4-31B-it-DFlash-GGUF

gemma-4-31B-it-DFlash-GGUF

GGUF conversion of z-lab/gemma-4-31B-it-DFlash for use with llama.cpp.

This is a DFlash draft model, not a standalone language model.

It must be used together with a compatible google/gemma-4-31B-it GGUF target model.

Model Details

• Source model: z-lab/gemma-4-31B-it-DFlash
• Compatible target: google/gemma-4-31B-it
• Format: GGUF
• Quantization: Q4_K_M
• Purpose: DFlash speculative decoding

This repository contains only the DFlash draft model. It does not include the Gemma 4 31B target model.

Compatibility

A recent version of llama.cpp with DFlash support is required.

Tested with:

• llama.cpp b9831
• NVIDIA GeForce RTX 5070 Ti 16 GB
• Target model: gemma-4-31B-it-UD-Q4_K_XL

Other llama.cpp builds, target quantizations, hardware configurations, prompts, and sampling settings may produce different performance.

Usage

llama-server

llama-server \
  -m gemma-4-31B-it-UD-Q4_K_XL.gguf \
  -md gemma-4-31B-it-DFlash-Q4_K_M.gguf \
  --spec-type draft-dflash \
  --spec-draft-n-max 4

llama-cli

llama-cli \
  -m gemma-4-31B-it-UD-Q4_K_XL.gguf \
  -md gemma-4-31B-it-DFlash-Q4_K_M.gguf \
  --spec-type draft-dflash \
  --spec-draft-n-max 4

The target and draft models must use the same tokenizer and vocabulary.

Suggested Settings

Based on the benchmark below:

• n_max = 4 gave the best aggregate wall-clock result.
• n_max = 2 gave the highest overall acceptance rate and performed better on several high-entropy natural-language tasks.
• n_max = 4–5 performed well for predictable outputs such as code completion, JSON, and repeated patterns.
• n_max = 5 was slower overall in this mixed benchmark, despite improving several structured tasks.

A reasonable general starting point is:

--spec-draft-n-max 4

For translation, creative writing, explanations, or conversational output, n_max = 2 may be preferable.

Conversion

Converted from the original Hugging Face DFlash checkpoint using convert_hf_to_gguf.py.

Gemma 4 tokenizer metadata was loaded from the compatible target model directory through --target-model-dir.

No neural-network weights were edited or retrained. The weights were converted and quantized to GGUF format.

Benchmark

Benchmarked using mtp-bench on an NVIDIA GeForce RTX 5070 Ti.

Target model:

gemma-4-31B-it-UD-Q4_K_XL

Draft model:

gemma-4-31B-it-DFlash-Q4_K_M

Aggregate Results

These results do not include a non-speculative baseline and therefore should not be interpreted as a direct speedup ratio over standard decoding.

Performance varies substantially by task type.

Detailed Results

n_max = 2

code_python        pred= 192 draft= 145 acc= 118 rate=0.814 tok/s=53.7
code_cpp           pred= 192 draft= 144 acc= 118 rate=0.819 tok/s=54.7
explain_concept    pred= 192 draft= 192 acc=  95 rate=0.495 tok/s=41.5
summarize          pred=  46 draft=  42 acc=  25 rate=0.595 tok/s=45.1
qa_factual         pred= 166 draft= 142 acc=  94 rate=0.662 tok/s=48.6
translation        pred=  16 draft=  18 acc=   7 rate=0.389 tok/s=36.5
creative_short     pred=  34 draft=  36 acc=  16 rate=0.444 tok/s=39.1
stepwise_math      pred= 192 draft= 148 acc= 117 rate=0.790 tok/s=53.5
json_output        pred= 192 draft= 132 acc= 125 rate=0.947 tok/s=60.2
long_reasoning     pred= 192 draft= 144 acc= 119 rate=0.826 tok/s=54.6
repeat_pattern     pred= 192 draft= 127 acc= 127 rate=1.000 tok/s=60.9
code_completion    pred= 192 draft= 136 acc= 123 rate=0.904 tok/s=58.0
long_code_review   pred= 192 draft= 182 acc=  99 rate=0.544 tok/s=42.3

Aggregate:
  requests:            13
  predicted tokens:    1990
  draft tokens:        1588
  accepted tokens:     1183
  acceptance rate:     0.7450
  total wall time:     42.37 s

n_max = 3

code_python        pred= 192 draft= 180 acc= 130 rate=0.722 tok/s=56.5
code_cpp           pred= 192 draft= 176 acc= 132 rate=0.750 tok/s=58.7
explain_concept    pred= 192 draft= 269 acc= 101 rate=0.376 tok/s=38.7
summarize          pred=  46 draft=  54 acc=  27 rate=0.500 tok/s=45.9
qa_factual         pred= 166 draft= 198 acc= 100 rate=0.505 tok/s=45.4
translation        pred=  16 draft=  27 acc=   7 rate=0.259 tok/s=31.7
creative_short     pred=  34 draft=  54 acc=  17 rate=0.315 tok/s=33.9
stepwise_math      pred= 192 draft= 183 acc= 130 rate=0.710 tok/s=56.7
json_output        pred= 192 draft= 152 acc= 140 rate=0.921 tok/s=67.3
long_reasoning     pred= 192 draft= 176 acc= 132 rate=0.750 tok/s=58.3
repeat_pattern     pred= 192 draft= 144 acc= 143 rate=0.993 tok/s=71.8
code_completion    pred= 192 draft= 159 acc= 137 rate=0.862 tok/s=63.5
long_code_review   pred= 192 draft= 260 acc= 104 rate=0.400 tok/s=38.7

Aggregate:
  requests:            13
  predicted tokens:    1990
  draft tokens:        2032
  accepted tokens:     1300
  acceptance rate:     0.6398
  total wall time:     41.36 s

n_max = 4

code_python        pred= 192 draft= 186 acc= 144 rate=0.774 tok/s=64.4
code_cpp           pred= 192 draft= 203 acc= 139 rate=0.685 tok/s=59.3
explain_concept    pred= 192 draft= 319 acc= 111 rate=0.348 tok/s=38.6
summarize          pred=  47 draft=  60 acc=  33 rate=0.550 tok/s=49.8
qa_factual         pred= 165 draft= 236 acc= 107 rate=0.453 tok/s=44.7
translation        pred=  16 draft=  36 acc=   7 rate=0.194 tok/s=28.2
creative_short     pred=  43 draft=  92 acc=  21 rate=0.228 tok/s=29.9
stepwise_math      pred= 192 draft= 204 acc= 140 rate=0.686 tok/s=60.0
json_output        pred= 192 draft= 168 acc= 149 rate=0.887 tok/s=71.6
long_reasoning     pred= 192 draft= 204 acc= 140 rate=0.686 tok/s=59.6
repeat_pattern     pred= 192 draft= 153 acc= 152 rate=0.994 tok/s=77.9
code_completion    pred= 192 draft= 180 acc= 146 rate=0.811 tok/s=66.9
long_code_review   pred= 192 draft= 291 acc= 116 rate=0.399 tok/s=40.1

Aggregate:
  requests:            13
  predicted tokens:    1999
  draft tokens:        2332
  accepted tokens:     1405
  acceptance rate:     0.6025
  total wall time:     40.39 s

n_max = 5

code_python        pred= 192 draft= 201 acc= 150 rate=0.746 tok/s=64.0
code_cpp           pred= 192 draft= 236 acc= 143 rate=0.606 tok/s=55.9
explain_concept    pred= 192 draft= 402 acc= 110 rate=0.274 tok/s=33.1
summarize          pred=  47 draft=  60 acc=  36 rate=0.600 tok/s=54.0
qa_factual         pred= 165 draft= 280 acc= 110 rate=0.393 tok/s=40.9
translation        pred=  16 draft=  45 acc=   7 rate=0.156 tok/s=24.6
creative_short     pred=  43 draft= 120 acc=  20 rate=0.167 tok/s=24.8
stepwise_math      pred= 192 draft= 222 acc= 146 rate=0.658 tok/s=59.5
json_output        pred= 192 draft= 187 acc= 153 rate=0.818 tok/s=69.6
long_reasoning     pred= 192 draft= 221 acc= 146 rate=0.661 tok/s=59.4
repeat_pattern     pred= 192 draft= 160 acc= 159 rate=0.994 tok/s=82.8
code_completion    pred= 192 draft= 182 acc= 154 rate=0.846 tok/s=71.7
long_code_review   pred= 192 draft= 394 acc= 110 rate=0.279 tok/s=32.7

Aggregate:
  requests:            13
  predicted tokens:    1999
  draft tokens:        2710
  accepted tokens:     1444
  acceptance rate:     0.5328
  total wall time:     43.02 s

Observations

• n_max = 2 achieved the highest aggregate draft acceptance rate.
• n_max = 4 achieved the shortest total wall time in this mixed benchmark.
• n_max = 5 improved highly predictable tasks such as repeated patterns and code completion, but increased total wall time.
• Translation, creative writing, conceptual explanations, and long-form code review showed substantially lower acceptance as n_max increased.
• Structured output, code completion, stepwise mathematics, and repeated patterns benefited more from longer draft sequences.

Notes

This repository contains only the DFlash draft model.

A compatible google/gemma-4-31B-it GGUF target model is required. The target GGUF may use a different quantization from the draft model, but both models must share compatible tokenizer and model architecture assumptions.

The benchmark results are specific to the tested hardware, model quantizations, llama.cpp build, prompts, and runtime settings.

Credits

• Z Lab — DFlash method and original draft checkpoint
• Google DeepMind — Gemma 4 and google/gemma-4-31B-it
• ggml-org — llama.cpp, GGUF, and DFlash inference support

License

This repository contains a converted and quantized GGUF version of the original DFlash draft checkpoint.

The upstream z-lab/gemma-4-31B-it-DFlash and google/gemma-4-31B-it repositories identify their applicable licensing terms. Users should review the upstream model cards and licenses before redistribution or commercial use.