sriq-diffusiongemma-26B-A4B-v1.6-FP8-dynamic

<div align="center"> <img src="https://huggingface.co/sriq-ai/sriq-diffusiongemma-26B-A4B-v1.6-FP8-dynamic/resolve/main/banner.png" width="100%" alt="SRIQ-DiffusionGemma-26B-A4B-V1.6 — compressed reasoning" /> </div> <p align="center"> <a href="https://sriq.org" target="_blank">SRIQ</a> | <a href="https://bench.sriq.org" target="_blank">Benchmarks</a> | <a href="https://huggingface.co/sriq-ai/sriq-diffusiongemma-26B-A4B-v1.6" target="_blank">bf16 model</a> | <a href="https://huggingface.co/datasets/sriq-ai/sriq-sft-v1.6" target="_blank">Training data</a> | <a href="https://huggingface.co/kaivoss/diffusiongemma-26B-A4B-it-ALWAYS-THINK" target="_blank">Base model</a> </p>

---

FP8 quantized build of sriq-ai/sriq-diffusiongemma-26B-A4B-v1.6. Weights and activations are FP8 (W8A8, dynamic per-token activation scales) in compressed-tensors format, produced with llm-compressor.

27.2 GB instead of 51.6 GB — it fits on one 40 GB accelerator, where the bf16 model needs 80 GB.

The model reasons in compressed Simplified Chinese and answers in the language it was asked in. DiffusionGemma is not autoregressive: a causal encoder reads the prompt into a read-only KV cache, and a bidirectional decoder denoises a 256-token canvas over ~48 refinement passes.

Quantization quality

20 held-out prompts (seeded random 5% split of v1.6's 9,988 rows), 384 max tokens, temperature 0, both models served through the same vLLM build so the comparison is apples to apples.

FP8 does not cost quality here — it scores marginally above bf16, which is noise at n=20, not an improvement.

These numbers are lower than the bf16 model card's 18.1%, and that is a harness difference, not a regression. The card's figure was measured through transformers with default sampling; this table is vLLM at temperature 0. Only compare rows within this table.

Accuracy of the reasoning itself is still not benchmarked, and the bf16 model's repetition limitation carries over — a minority of prompts collapse into a repeated phrase.

Deployment

docker pull vllm/vllm-openai:gemma

docker run -itd --name sriq-diffusiongemma-fp8 \
    --ipc=host --network host --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -e VLLM_USE_V2_MODEL_RUNNER=1 \
    vllm/vllm-openai:gemma \
        --model sriq-ai/sriq-diffusiongemma-26B-A4B-v1.6-FP8-dynamic \
        --max-num-seqs 4 \
        --gpu-memory-utilization 0.85 \
        --host 0.0.0.0 --port 8000

Requires vLLM 0.24.0+ and Hopper/Ada or newer — FP8 tensor cores. On Ampere (A100), vLLM's Marlin FP8 MoE kernel cannot tile this expert shape within the available shared memory; quantize to INT8 W8A8 instead, which routes through the Triton int8 MoE backend.

Keep --max-num-seqs low: diffusion state buffers pre-allocate max_seqs × canvas_length × vocab_size, so concurrency costs VRAM in a way ordinary autoregressive serving does not.

SGLang — not supported yet

SGLang cannot serve DiffusionGemma. --dllm-algorithm Gemma4Renoise parses, which makes it look supported, but the flag value is not a registered algorithm and the architecture is not a registered model:

RuntimeError: Unknown diffusion LLM: DiffusionGemmaForBlockDiffusion

Verified on lmsysorg/sglang:latest (v0.5.20). The dLLM registry in srt/dllm/config.py holds only LLaDA2MoeModelLM, SDARForCausalLM and SDARMoeForCausalLM — on the released image and on main — and srt/dllm/algorithm/ contains no renoise sampler. This is not specific to these weights: Google's own checkpoint declares the same architectures.

Support is in PR #34061, still open, and it adds bf16 serving only. The LMSYS cookbook page documents the command ahead of the code. Use vLLM until that lands.

Creation

Scripts are quantize.py and dedupe.py. The recipe follows RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic:

QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=["lm_head", "re:.*embed.*", "re:.*router",
            "re:.*vision_tower.*", "re:.*self_conditioning.*"],
)

FP8_DYNAMIC needs no calibration dataset — weight scales are computed from the weights, activation scales at runtime. Two things still have to be handled by hand:

The MoE experts are a fused 3D tensor. 88% of the parameters live in layers.N.experts.{gate_up,down}_proj, which targets="Linear" cannot see — left alone, the quantization would skip almost the whole model. llmcompressor.modeling.moe.linearize_moe unpacks them into per-expert nn.Linear first, permanently, which is also the layout vLLM's loader expects.

Encoder and decoder share tied weights, and both steps break the tie. Linearizing renames the experts, and FP8 conversion replaces each weight with a fresh parameter, so save_pretrained no longer matches its _tied_weights_keys patterns and writes the encoder and decoder views of one shared storage as two separate tensors — 51.7 GB instead of 27.2 GB, with every expert stored twice. dedupe.py drops the duplicated encoder tensors afterwards, using the bf16 checkpoint's own key set to decide which encoder keys are tied. Value equality is not a safe test: the per-side layer_scalar tensors happen to hold equal numbers but are genuinely separate parameters.

The result is key-for-key identical to Red Hat's FP8 build of the base model (24,232 tensors, 27.2 GB).

See FINETUNE.md on the bf16 repo for how the model itself was trained.