gemma4-emirati-family-chatbot-lora

Gemma 4 E4B - Emirati Family Chatbot (LoRA Adapter)

A QLoRA fine-tuned adapter for google/gemma-4-E4B-it, specialized in family advice in Emirati Arabic dialect.

What it does

• Responds to family-related questions in Emirati Arabic dialect (not MSA/formal Arabic)
• Covers: parenting, cooking, traditions, adolescent topics, celebrations, daily life
• Politely refuses out-of-scope topics (programming, investments, etc.)
• Maintains a warm, professional assistant persona (never pretends to be a family member)

Training Details

Usage

With PEFT

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained("google/gemma-4-E4B-it", torch_dtype=torch.float16)
tokenizer = AutoTokenizer.from_pretrained("google/gemma-4-E4B-it")

# IMPORTANT: Unwrap Gemma4ClippableLinear before loading adapter
from transformers.models.gemma4.modeling_gemma4 import Gemma4ClippableLinear
for name, module in base_model.named_modules():
    for attr_name, child in list(module.named_children()):
        if isinstance(child, Gemma4ClippableLinear):
            setattr(module, attr_name, child.linear)

# Load adapter
model = PeftModel.from_pretrained(base_model, "Aledec/gemma4-emirati-family-chatbot-lora")

With llama.cpp (recommended for inference)

For fastest inference, fuse the adapter with the base model and quantize to GGUF:

# Fuse adapter
python scripts/fuse_model.py \
    --base-model google/gemma-4-E4B-it \
    --adapter-path Aledec/gemma4-emirati-family-chatbot-lora \
    --output-path models/fused/gemma4-v1

# Convert and quantize
python llama.cpp/convert_hf_to_gguf.py models/fused/gemma4-v1 --outfile gemma4-f16.gguf --outtype f16
llama.cpp/build/bin/llama-quantize gemma4-f16.gguf gemma4-q5km.gguf Q5_K_M

# Run server
llama.cpp/build/bin/llama-server -m gemma4-q5km.gguf -t 4 -c 2048 --port 8091

Performance (Q5_K_M GGUF on M3 Max)

Technical Note: Gemma4ClippableLinear

Gemma 4 is multimodal (vision + audio + text). The vision/audio tower layers use Gemma4ClippableLinear (a wrapper around nn.Linear with input/output clipping) which PEFT does not support natively. You must unwrap these layers to their inner nn.Linear before loading the adapter. See the usage example above.

Links

• Training repo: AleDeclerk/ExperimentalFamilyChatbot
• Full model comparison: MODEL_COMPARISON.md

License

MIT