Gemma-4-12B-MXFP4A16-Text-Only

Gemma-4-12B-MXFP4A16-Text-Only

This model is an experimental MXFP4A16 compressed checkpoint derived directly from google/gemma-4-12B-it.

It was quantized from the original BF16 base model, not from the FP8 model. Likely image/audio special-token embeddings were zeroed, and the model is intended to be served as text-only.

Model Details

Text-Only Changes

This checkpoint disables likely image/audio special-token embeddings by zeroing their embedding rows. The tokenizer entries and architecture remain present so the model can still load with Gemma 4 tooling.

Serve with multimodal inputs blocked:

limit_mm_per_prompt={"image": 0, "audio": 0}

For vLLM, ensure the checkpoint includes:

processor_config.json

Quantization Recipe

Default scheme:

MXFP4A16

Targeted layers:

Linear

Ignored layers/patterns:

lm_head
re:.*embed.*
re:.*router.*
re:.*vision.*
re:.*audio.*
re:.*multimodal.*
re:.*image.*

Intended Use

This checkpoint is intended for:

• text-only generation
• context-grounded QA
• small-footprint serving experiments
• comparison against the FP8 checkpoint

It is not intended for image/audio input.

Usage With vLLM

vllm serve /path/to/model \
  --served-model-name fp4-gemma \
  --quantization compressed-tensors \
  --dtype bfloat16 \
  --max-model-len 2048 \
  --gpu-memory-utilization 0.45 \
  --enforce-eager

HotpotQA Benchmark

Benchmark:

Dataset: HotpotQA Distractor
Slice: validation[0:459]
Model name: fp4-gemma
Successful requests: 459/500

System prompt:

Answer questions using only the provided context. Some questions require combining facts from multiple passages. Return only the shortest correct answer. When possible, copy the exact name, date, place, number, or phrase from the context. Do not explain your reasoning or add extra text.

Limitations

• Experimental text-only quantized checkpoint.
• Serve with multimodal inputs blocked.
• Reported benchmark is HotpotQA Distractor validation[0:459].

License

This checkpoint is derived from google/gemma-4-12B-it. Follow the applicable Gemma 4 terms and Apache 2.0 license information from the base model.