Downloads · 30 days
131
3% of all-time downloads
Danny-Dasilva/gemma-4-12B-it-W4A16-GPTQ-g32-DSpark
gemma-4-12B-it-W4A16-GPTQ-g32-DSpark is a text generation model from Danny-Dasilva. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as gemma.
A post-training GPTQ int4 (W4A16, groupsize=32, symmetric) quantization of google/gemma-4-12B-it, built specifically to be a DSpark speculative-decoding target (draft head: deepseek-ai/dsparkgemma412bblock7).
Downloads · 30 days
131
3% of all-time downloads
All-time downloads
4K
Public
Parameters
12.3B
8.3 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors8.3 GB · 100%
How the weights are stored.
I3210.9B · 89%
From the Hugging Face model README
A post-training GPTQ int4 (W4A16, group_size=32, symmetric) quantization of
google/gemma-4-12B-it, built specifically to be a
DSpark speculative-decoding target
(draft head: deepseek-ai/dspark_gemma4_12b_block7).
TL;DR: 198 tok/s single-stream on one RTX 5090 (32 GB) — the fastest gemma-4-12B configuration we measured on that card, at 7.8 GB of weights.
The DSpark draft head was trained against the bf16 gemma-4-12B-it weights. Any retraining of the
target — including the official QAT checkpoint
google/gemma-4-12B-it-qat-w4a16-ct —
drifts far enough that draft acceptance collapses and speculation makes the model slower:
| target (same W4A16 format) | DSpark per-token accept | outcome |
|---|---|---|
| QAT w4a16-ct (retrained) | 2.1% | slower than no speculation |
| this model (post-training GPTQ) | 35.6% | 1.73× speedup |
| fp8 (post-training, reference) | 34.4% | 1.9× over its own native |
| bf16 (reference) | 39.5% | 2.7× (eager) |
Post-training rounding preserves draft alignment; QAT retraining destroys it. If you pair gemma-4-12B int4 with the DeepSeek DSpark draft, you need a post-training quant like this one.
vLLM (recent main with gemma-4 DSpark support), FlashInfer target attention, greedy, 256-token generations over 4 chat prompts, CUDA graphs (piecewise):
| config | tok/s | vs native int4 |
|---|---|---|
| this model + DSpark k=7 + CUDA graphs + FlashInfer autotune | 198.2 (peaks 239) | 1.73× |
| this model + DSpark k=7 + CUDA graphs | 193.8 | 1.69× |
| this model + DSpark k=14 (2 draft blocks) | 178.6 | don't — accept dies at block boundary |
| this model, native (no speculation), CUDA graphs | 114.7 | 1.0× |
| fp8 (online) + DSpark k=7 (13 GB weights) | 165 | — |
| bf16 + DSpark k=7 (eager; graphs OOM on 32 GB) | 144 | — |
| official QAT w4a16-ct, native | 118 | speculation hurts this one |
| llama.cpp Q4_0 GGUF, native (reference) | 156 | its DSpark port is slower than native |
Weights: 7.8 GB (vs 24 GB bf16, 13 GB fp8) — target + draft + KV fit comfortably in 32 GB with CUDA graphs, which bf16 cannot do.
from vllm import LLM, SamplingParams
from transformers import AutoTokenizer
MODEL = "Danny-Dasilva/gemma-4-12B-it-W4A16-GPTQ-g32-DSpark"
DRAFT = "deepseek-ai/dspark_gemma4_12b_block7"
llm = LLM(
model=MODEL,
max_model_len=8192,
attention_backend="FLASHINFER", # gemma-4 full-attn layers have head_dim 512
enable_flashinfer_autotune=True, # +2% in our runs
speculative_config={
"method": "dspark",
"model": DRAFT,
"num_speculative_tokens": 7, # = draft block size; k<7 or k=14 are both slower
"attention_backend": "TRITON_ATTN",
},
)
tok = AutoTokenizer.from_pretrained(MODEL)
prompt = tok.apply_chat_template(
[{"role": "user", "content": "Explain transformers step by step."}],
add_generation_prompt=True, tokenize=False,
)
print(llm.generate([prompt], SamplingParams(temperature=0.0, max_tokens=256))[0].outputs[0].text)
Notes:
<|turn>-style tokens and
add_bos_token=False; raw prompts produce garbage on any backend.num_speculative_tokens must be ≤ 7 or a multiple of 7 (draft block size). 7 is optimal.kv_cache_dtype="fp8" measured slower (−3%) at short context on this card.llm-compressor 0.12 GPTQ, one-shot:
HuggingFaceH4/ultrachat_200k (train_sft), chat template applied,
max_seq_length 2048pack-quantized format —
mirrors the official QAT checkpoint's config so every vLLM kernel path is identicalsequential_targets=["Gemma4UnifiedTextDecoderLayer"]; lm_head and vision/audio embedder
projections kept in bf16 (same 17-entry ignore list as the official QAT release, spelled so both
HF and vLLM module names match)processor_config.json included (vLLM's multimodal processor init requires it)google/gemma-4-12B-it @ 5926caa4 (bf16, untouched — no finetuning)