gemma4-31b-qat-text-only

Gemma 4 31B IT QAT — text-only bf16 GGUF

A single-file bf16 GGUF of the language model only from Google's Gemma 4 31B instruction-tuned quantization-aware-trained (QAT) checkpoint. The vision and audio towers are removed and the tensors are laid out as a standalone Gemma4ForCausalLM, so standard GGUF tooling (llama.cpp, Ollama, LM Studio, …) can load it directly — and, more importantly, re-quantize it (e.g. ollama create --quantize q4_K_M).

Why this exists

The Gemma 4 omni releases ship as a Gemma4ForConditionalGeneration checkpoint with the language model nested under a language_model.* prefix beside vision/audio towers. ollama create --quantize cannot convert that layout — its bundled converter does not split the language model out the way the official gemma4:31b-it-qat release does (LM GGUF + separate mmproj GGUF), so conversion fails to find token_embd.weight and the resulting tag does not load. This repo publishes the working intermediate: a clean, text-only bf16 GGUF you can quantize to any scheme llama.cpp supports.

How it was made

1. Source checkpoint: mlx-community/gemma-4-31B-it-qat-bf16 — the quantization-aware-trained weights in bf16, as the full omni checkpoint. No dequantization round trip is involved.

2. Text-only extraction with extract_gemma4_text.py, included in this repo (uses Apple-silicon MLX): keeps only language_model.* tensors, renames language_model.model.X → model.X, rewrites config.json from text_config with architectures = ["Gemma4ForCausalLM"] (carrying over eos_token_id and transformers_version), copies tokenizer.json, tokenizer_config.json, generation_config.json, and chat_template.jinja, and rebuilds model.safetensors.index.json for the surviving tensors.

3. GGUF conversion with a current llama.cpp checkout (older packaged builds may lack the converter or Gemma 4 support):

   python convert_hf_to_gguf.py <text_only_dir> \
       --outfile gemma4-31b-qat-text-only.gguf --outtype bf16
   

Usage

The main intended use is q4_K_M re-quantization, motivated by Huang & Wang (IJCNN 2025), LLMs at the Edge: Performance and Efficiency Evaluation with Ollama on Diverse Hardware, which found q4_K_M builds matching or exceeding bf16 task performance for open-weight models served through Ollama:

printf 'FROM ./gemma4-31b-qat-text-only.gguf\n' > Modelfile
ollama create gemma4:31b-it-qat-text-only -f Modelfile --quantize q4_K_M
ollama run gemma4:31b-it-qat-text-only

Or run directly with llama.cpp (bf16 needs ~62 GB of memory; quantize first for less):

llama-cli -m gemma4-31b-qat-text-only.gguf -p "Hello"

Caveats

• Text in, text out only. The vision and audio towers are not included; multimodal prompts will not work. Use the official omni releases for that.
• Not bit-identical to the official QAT GGUF. This file is converted from the bf16 QAT weights along a different path than Google's official it-qat GGUF artifact. In our evaluations a q4_K_M re-quant of this file scored within noise of the vendor QAT artifact on our task, but no equivalence is claimed.
• Gemma 4 is released under Apache 2.0; this derivative carries the same license.