diffusiongemma-26B-A4B-iq2xxs-4070

DiffusionGemma 26B-A4B IQ2 — pack for RTX 4070 (12 GB)

Ready-to-run IQ2_XXS GGUF of Google’s DiffusionGemma 26B-A4B plus two llama.cpp patches that cut ~2.7 GiB of waste so a desktop 4070 12 GB can run it.

Not stock llama.cpp. You need PR #24423 (llama-diffusion-cli) and patches/4070-vram.patch from this repo. Unpatched PR peaks ~13.8 GiB and will OOM a 12 GB card.

---

Quick start (Linux + NVIDIA)

1. Download this repo

# huggingface-cli
hf download Infatoshi/diffusiongemma-26B-A4B-iq2xxs-4070 --local-dir diffusiongemma-4070
cd diffusiongemma-4070

2. Build the patched diffusion runner

# CUDA toolkit required
./scripts/build_llama_diffusion_4070.sh ~/src/llama.cpp-diffusion-4070
export PATH="$HOME/src/llama.cpp-diffusion-4070/build/bin:$PATH"
# or: export CLI=$HOME/src/llama.cpp-diffusion-4070/build/bin/llama-diffusion-cli

Manual equivalent:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git fetch origin pull/24423/head:diffusiongemma && git checkout diffusiongemma
git apply /path/to/this-repo/patches/4070-vram.patch
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc) --target llama-diffusion-cli

3. Run on a 4070

# Desktop GPU that also drives the display (recommended default)
export CLI=~/src/llama.cpp-diffusion-4070/build/bin/llama-diffusion-cli
export N_CPU_MOE=4
./scripts/run_4070.sh "Why is the sky blue? Answer in two short sentences."

Or one-liner:

llama-diffusion-cli \
  -m diffusiongemma-26B-A4B-iq2xxs.gguf \
  -ngl 99 --n-cpu-moe 4 -n 256 -ub 1 -b 1 \
  --diffusion-eb auto --perf \
  -p "Your prompt here"

System RAM: leave ≥16 GB free for the few MoE layers on CPU (n-cpu-moe).

---

What the patches do

1. Ghost ubatch fix (diffusion-cli.cpp)
   Stock PR sizes n_ubatch = canvas + 2048 even for one-shot prompts → ~2 GiB wasted fp32 logits at vocab 262k. One-shot runs size to the real tokenized prompt instead.

2. Self-cond embedding Q8 (diffusion-gemma.cpp)
   Device SC embedding table was a full F16 dequant copy (~1.4 GiB); store as Q8_0 (~0.7 GiB). Quality checked vs F16 SC.

Together with --n-cpu-moe 3–4, peak drops from ~13.8 GiB → ~10.4 GiB.

---

Measured numbers (anvil RTX 3090, same flags; 4070 not in fleet)

• Full experts on GPU still need ~11.4 GiB after patches → over a clean 10.5 GiB desktop budget.
• 4070 bandwidth is lower than 3090; expect roughly half full-GPU speed. Offloaded experts also depend on your CPU/DDR.

Quality (50%-mask pseudo-ppl vs Unsloth Q4_K_M): IQ2 within ~9% of Q4 — usable for chat/coding smoke; not near-BF16.

---

Longer prompts / more tokens

• Each extra ~260 prompt tokens grows ubatch; add about +1 n-cpu-moe.
• -n 512 (two canvases): use N_CPU_MOE=5.
• Interactive multi-turn still uses larger headroom (conversation mode keeps slack).

---

Attribution

This repo does not claim the quant methodology; we only package weights + runtime instructions that fit 12 GB after measurement.

---

Troubleshooting

---

Files

diffusiongemma-26B-A4B-iq2xxs.gguf   # weights (~9.4 GiB)
patches/4070-vram.patch              # required llama.cpp changes
scripts/build_llama_diffusion_4070.sh
scripts/run_4070.sh
README.md