gemma-4-26B-A4B-it-DFlash-GGUF

gemma-4-26B-A4B-it-DFlash-GGUF

GGUF conversion of z-lab/gemma-4-26B-A4B-it-DFlash for use with llama.cpp.

This is a DFlash draft model, not a standalone language model.

It must be used together with a compatible google/gemma-4-26B-A4B-it GGUF target model.

Model Details

• Source model: z-lab/gemma-4-26B-A4B-it-DFlash
• Compatible target: google/gemma-4-26B-A4B-it
• Format: GGUF
• Quantization: Q4_K_M
• Purpose: DFlash speculative decoding

This repository contains only the DFlash draft model. It does not include the Gemma 4 26B-A4B target model.

Compatibility

A recent version of llama.cpp with DFlash support is required.

Tested with:

• llama.cpp b9831
• NVIDIA GeForce RTX 5070 Ti 16 GB
• Target model: gemma-4-26B-A4B-it-UD-Q4_K_XL

Other llama.cpp builds, target quantizations, hardware configurations, prompts, and sampling settings may produce different performance.

Usage

llama-server

llama-server \
  -m gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf \
  -md gemma-4-26B-A4B-it-DFlash-Q4_K_M.gguf \
  --spec-type draft-dflash \
  --spec-draft-n-max 3

llama-cli

llama-cli \
  -m gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf \
  -md gemma-4-26B-A4B-it-DFlash-Q4_K_M.gguf \
  --spec-type draft-dflash \
  --spec-draft-n-max 3

The target and draft models must use the same tokenizer and vocabulary.

Suggested Settings

Based on the benchmark below:

• n_max = 3 gave the best aggregate wall-clock result.
• n_max = 2 gave the highest overall acceptance rate and performed better on several high-entropy natural-language tasks.
• n_max = 4–5 performed well for predictable outputs such as code completion, JSON, and repeated patterns.
• n_max = 5 improved several highly structured tasks, but was slower overall than n_max = 3 in this mixed benchmark.

A reasonable general starting point is:

--spec-draft-n-max 3

For translation, creative writing, explanations, or conversational output, n_max = 2 may be preferable. For code completion, JSON, and highly predictable output, n_max = 4–5 may perform better.

Conversion

Converted from the original Hugging Face DFlash checkpoint using convert_hf_to_gguf.py.

Gemma 4 tokenizer metadata was loaded from the compatible target model directory through --target-model-dir.

No neural-network weights were edited or retrained. The weights were converted and quantized to GGUF format.

Benchmark

Benchmarked using mtp-bench on an NVIDIA GeForce RTX 5070 Ti.

Target model:

gemma-4-26B-A4B-it-UD-Q4_K_XL

Draft model:

gemma-4-26B-A4B-it-DFlash-Q4_K_M

Aggregate Results

These results do not include a non-speculative baseline and therefore should not be interpreted as a direct speedup ratio over standard decoding.

Performance varies substantially by task type.

Detailed Results

n_max = 2

code_python        pred= 192 draft= 149 acc= 116 rate=0.778 tok/s=147.8
code_cpp           pred= 192 draft= 158 acc= 111 rate=0.703 tok/s=146.0
explain_concept    pred= 192 draft= 195 acc=  92 rate=0.472 tok/s=118.0
summarize          pred=  51 draft=  44 acc=  29 rate=0.659 tok/s=138.2
qa_factual         pred= 192 draft= 180 acc= 101 rate=0.561 tok/s=130.5
translation        pred=  26 draft=  30 acc=  11 rate=0.367 tok/s=105.3
creative_short     pred=  41 draft=  46 acc=  17 rate=0.370 tok/s=108.6
stepwise_math      pred= 192 draft= 158 acc= 112 rate=0.709 tok/s=146.3
json_output        pred= 192 draft= 136 acc= 123 rate=0.904 tok/s=167.1
long_reasoning     pred= 192 draft= 155 acc= 112 rate=0.723 tok/s=145.7
repeat_pattern     pred= 192 draft= 132 acc= 125 rate=0.947 tok/s=175.7
code_completion    pred= 192 draft= 135 acc= 123 rate=0.911 tok/s=169.0
long_code_review   pred= 192 draft= 188 acc=  96 rate=0.511 tok/s=119.4

Aggregate:
  requests:            13
  predicted tokens:    2038
  draft tokens:        1706
  accepted tokens:     1168
  acceptance rate:     0.6846
  total wall time:     16.28 s

n_max = 3

code_python        pred= 192 draft= 197 acc= 125 rate=0.634 tok/s=156.2
code_cpp           pred= 192 draft= 211 acc= 120 rate=0.569 tok/s=149.6
explain_concept    pred= 192 draft= 283 acc=  95 rate=0.336 tok/s=110.1
summarize          pred=  51 draft=  57 acc=  31 rate=0.544 tok/s=143.9
qa_factual         pred= 189 draft= 225 acc= 114 rate=0.507 tok/s=139.8
translation        pred=  26 draft=  39 acc=  12 rate=0.308 tok/s=108.8
creative_short     pred=  37 draft=  63 acc=  16 rate=0.254 tok/s=97.5
stepwise_math      pred= 192 draft= 195 acc= 125 rate=0.641 tok/s=159.4
json_output        pred= 192 draft= 155 acc= 139 rate=0.897 tok/s=196.9
long_reasoning     pred= 192 draft= 198 acc= 124 rate=0.626 tok/s=153.9
repeat_pattern     pred= 192 draft= 150 acc= 141 rate=0.940 tok/s=210.0
code_completion    pred= 192 draft= 150 acc= 141 rate=0.940 tok/s=207.9
long_code_review   pred= 192 draft= 276 acc=  98 rate=0.355 tok/s=110.1

Aggregate:
  requests:            13
  predicted tokens:    2031
  draft tokens:        2199
  accepted tokens:     1281
  acceptance rate:     0.5825
  total wall time:     15.29 s

n_max = 4

code_python        pred= 192 draft= 226 acc= 134 rate=0.593 tok/s=161.6
code_cpp           pred= 192 draft= 255 acc= 127 rate=0.498 tok/s=151.7
explain_concept    pred= 192 draft= 348 acc= 103 rate=0.296 tok/s=110.5
summarize          pred=  51 draft=  72 acc=  33 rate=0.458 tok/s=139.8
qa_factual         pred= 187 draft= 284 acc= 116 rate=0.408 tok/s=134.7
translation        pred=  26 draft=  52 acc=  13 rate=0.250 tok/s=101.9
creative_short     pred=  42 draft=  92 acc=  19 rate=0.206 tok/s=92.6
stepwise_math      pred= 192 draft= 233 acc= 132 rate=0.567 tok/s=162.0
json_output        pred= 192 draft= 172 acc= 148 rate=0.861 tok/s=215.3
long_reasoning     pred= 192 draft= 272 acc= 122 rate=0.449 tok/s=138.2
repeat_pattern     pred= 192 draft= 160 acc= 151 rate=0.944 tok/s=239.1
code_completion    pred= 192 draft= 172 acc= 148 rate=0.861 tok/s=221.4
long_code_review   pred= 192 draft= 376 acc=  96 rate=0.255 tok/s=100.6

Aggregate:
  requests:            13
  predicted tokens:    2034
  draft tokens:        2714
  accepted tokens:     1342
  acceptance rate:     0.4945
  total wall time:     15.40 s

n_max = 5

code_python        pred= 192 draft= 255 acc= 139 rate=0.545 tok/s=163.9
code_cpp           pred= 192 draft= 302 acc= 130 rate=0.430 tok/s=148.9
explain_concept    pred= 192 draft= 434 acc= 103 rate=0.237 tok/s=102.7
summarize          pred=  51 draft=  85 acc=  34 rate=0.400 tok/s=137.6
qa_factual         pred= 187 draft= 345 acc= 118 rate=0.342 tok/s=128.4
translation        pred=  26 draft=  65 acc=  13 rate=0.200 tok/s=93.8
creative_short     pred=  42 draft= 115 acc=  19 rate=0.165 tok/s=85.9
stepwise_math      pred= 192 draft= 280 acc= 134 rate=0.479 tok/s=157.4
json_output        pred= 192 draft= 187 acc= 153 rate=0.818 tok/s=228.9
long_reasoning     pred= 192 draft= 323 acc= 126 rate=0.390 tok/s=137.3
repeat_pattern     pred= 192 draft= 177 acc= 155 rate=0.876 tok/s=244.7
code_completion    pred= 192 draft= 193 acc= 152 rate=0.788 tok/s=227.3
long_code_review   pred= 192 draft= 454 acc= 100 rate=0.220 tok/s=97.0

Aggregate:
  requests:            13
  predicted tokens:    2034
  draft tokens:        3215
  accepted tokens:     1376
  acceptance rate:     0.4280
  total wall time:     15.66 s

Observations

• n_max = 2 achieved the highest aggregate draft acceptance rate.
• n_max = 3 achieved the shortest total wall time in this mixed benchmark.
• n_max = 4 was nearly tied with n_max = 3, but used substantially more draft tokens.
• n_max = 5 improved highly predictable tasks such as repeated patterns, JSON output, and code completion, but increased total wall time.
• Translation, creative writing, conceptual explanations, and long-form code review showed substantially lower acceptance as n_max increased.
• Structured output and code completion benefited more from longer draft sequences.

Notes

This repository contains only the DFlash draft model.

A compatible google/gemma-4-26B-A4B-it GGUF target model is required. The target GGUF may use a different quantization from the draft model, but both models must share compatible tokenizer and model architecture assumptions.

The benchmark results are specific to the tested hardware, model quantizations, llama.cpp build, prompts, and runtime settings.

Credits

• Z Lab — DFlash method and original draft checkpoint
• Google DeepMind — Gemma 4 and google/gemma-4-26B-A4B-it
• ggml-org — llama.cpp, GGUF, and DFlash inference support

License

This repository contains a converted and quantized GGUF version of the original DFlash draft checkpoint.

The upstream z-lab/gemma-4-26B-A4B-it-DFlash and google/gemma-4-26B-A4B-it repositories identify their applicable licensing terms. Users should review the upstream model cards and licenses before redistribution or commercial use.