Gemma-4-E4B-GLM5.1-distill

Gemma 4 E4B — GLM-5.1 Reasoning Fine-tune

A full SFT fine-tune of google/gemma-4-E4B-it on the complete open-source GLM-5.1-1000000x dataset (~614k conversations), trained with LoRA + DDP across 6× A100-40GB GPUs.

This is a merged model — no PEFT or adapter files needed. Load it directly like any standard HuggingFace model.

The model produces structured, step-by-step chain-of-thought reasoning in the GLM-5.1 style, covering multi-step mathematics, PhD-level science, and multilingual STEM problems.

---

Quick Start — Deploy with vLLM

The recommended way to run this model is via vLLM, which gives you an OpenAI-compatible API server with full 128k context support.

1. Install vLLM

pip install vllm

2. Start the server

vllm serve Dhiaul/Gemma-4-E4B-GLM5.1-distill \
  --dtype bfloat16 \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.92 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 8192 \
  --served-model-name gemma4-glm \
  --host 0.0.0.0 \
  --port 8000

GPU memory: A single A100-40GB or H100-80GB is enough. For concurrency beyond ~5 simultaneous 128k requests, add --tensor-parallel-size 2 and two GPUs.

3. Send a request

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma4-glm",
    "messages": [
      {"role": "user", "content": "Prove that sqrt(2) is irrational, step by step."}
    ],
    "max_tokens": 2048
  }'

Python client

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")

response = client.chat.completions.create(
    model="gemma4-glm",
    messages=[{
        "role": "user",
        "content": "Solve: A train travels 120 km at 60 km/h, then 80 km at 40 km/h. What is the average speed for the whole journey? Show all steps."
    }],
    max_tokens=1024,
    temperature=1.0,
    top_p=0.95,
)
print(response.choices[0].message.content)

---

Load with Transformers (no vLLM)

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Dhiaul/Gemma-4-E4B-GLM5.1-distill"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [{"role": "user", "content": "If f(x) = 3x² - 2x + 1, find f'(x) and explain each step."}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=1024,
        do_sample=True,
        temperature=1.0,
        top_p=0.95,
    )

response = outputs[0][inputs.input_ids.shape[1]:]
print(tokenizer.decode(response, skip_special_tokens=True))

---

vLLM Configuration Reference

Minimum requirements for vLLM:

• GPU: 1× GPU with ≥24 GB VRAM (tested on A100-40GB)
• vLLM ≥ 0.7 (Gemma 4 support)
• --enable-chunked-prefill is required for long-context (128k) prompts — without it, large prefills OOM

---

Model Details

---

Training Dataset

GLM-5.1-1000000x is an open-source (Apache 2.0) dataset of distilled long-form reasoning traces in the GLM style. We used all three available subsets:

Preprocessing applied:

• Gemma 4 chat template (tokenizer.apply_chat_template)
• Filtered sequences over 8,192 tokens (~2% of samples dropped)
• Stratified 95/5 train/val split across subsets

---

Training Configuration

LoRA Config (before merge)

LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    # Gemma 4 vision/audio towers use Gemma4ClippableLinear — not LoRA-injectable
    exclude_modules=r".*(?:vision_tower|audio_tower|embed_vision|embed_audio).*",
    bias="none",
    task_type="CAUSAL_LM",
)

Key Training Arguments

SFTConfig(
    num_train_epochs=1,
    per_device_train_batch_size=1,
    per_device_eval_batch_size=1,
    gradient_accumulation_steps=8,   # effective batch = 1 × 8 × 6 GPUs = 48
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    max_length=2048,
    bf16=True,
    gradient_checkpointing=True,
    optim="adamw_torch_fused",
    ddp_find_unused_parameters=False,
)

Framework versions: TRL 1.3.0 · PEFT 0.19.1 · Transformers 5.6.2 · Accelerate 1.13.0 · PyTorch 2.11.0

---

Limitations

• Sequence length during training: Each sequence was capped at 2,048 tokens during training. The base model's positional encoding is preserved through LoRA, so vLLM can serve at 128k context — but performance on very long reasoning chains was not formally evaluated.
• All-token training: We trained on all tokens (user + assistant turns), not assistant-only. This is slightly less sample-efficient but does not affect reasoning quality in practice.
• Model size ceiling: E4B is a small-but-capable model. Its absolute reasoning ceiling is lower than 26B/70B models. Use this as a cost-efficient reasoning model rather than a SOTA benchmark.
• Language: Primarily English. The Multilingual-STEM subset adds partial coverage of other languages, but English strongly dominates the training distribution.
• Safety: The base Gemma 4 safety alignment is largely preserved through LoRA fine-tuning. However, this fine-tune was not independently red-teamed — evaluate for your use case before production deployment.
• Multimodal inputs: The base Gemma 4 vision/audio towers were not fine-tuned (excluded from LoRA). Multimodal input may produce less aligned outputs.

---

Reproduce This Fine-Tune

Full code and step-by-step instructions: github.com/Dhiaul/Gemma-4-E4B-GLM5.1-distill

Key non-obvious detail for multi-GPU reproduction: pre-tokenize your parquet files before training (scripts/tokenize_parquet.py). TRL's dataset preparation runs inside a main_process_first() barrier — rank 0 tokenizes all 614k rows while other ranks wait at an NCCL barrier. Without pre-tokenization this takes ~30 minutes and hits PyTorch's 1800s NCCL timeout, killing the job silently.

---

License

This fine-tune inherits the Gemma license from the base model. The training dataset is Apache 2.0.

---

Acknowledgements

• Google DeepMind for Gemma 4
• Kassadin88 for the GLM-5.1-1000000x open dataset
• HuggingFace TRL, PEFT, and vLLM teams