gemma-4-31b-it-merged-16bit-checkpoint-10700

gemma-4-31b-it-merged-16bit-checkpoint-10700

Merged 16-bit (bfloat16) checkpoint of unsloth/gemma-4-31B-it with LoRA adapters from training step 10700.

LoRA is baked into the base weights (merge_and_unload). This is a full Gemma4ForConditionalGeneration checkpoint and can be served with vLLM without --enable-lora.

Source adapters: /workspace/llm-finetune/checkpoint-10700.

Validation loss (step 10700)

Serve with vLLM

Requires a vLLM build that includes Gemma 4 (vLLM ≥ 0.19). 31B bf16 needs about 62 GB of VRAM (one 80 GB GPU, or tensor parallel across two).

vllm serve vaghawan/gemma-4-31b-it-merged-16bit-checkpoint-10700 \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4

Multi-GPU:

vllm serve vaghawan/gemma-4-31b-it-merged-16bit-checkpoint-10700 \
  --dtype bfloat16 \
  --tensor-parallel-size 2 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4

Load with Transformers

from transformers import AutoProcessor, Gemma4ForConditionalGeneration

model = Gemma4ForConditionalGeneration.from_pretrained(
    "vaghawan/gemma-4-31b-it-merged-16bit-checkpoint-10700",
    torch_dtype="bfloat16",
    device_map="auto",
)
processor = AutoProcessor.from_pretrained("vaghawan/gemma-4-31b-it-merged-16bit-checkpoint-10700")