gemma-3-4b-it-tulumix-30k

gemma-3-4b-it-tulumix-30k

A supervised fine-tune of google/gemma-3-4b-it on a 30,000-conversation stratified sample of allenai/tulu-3-sft-mixture, trained with QLoRA and merged back into full bf16 weights.

This run was designed to test whether mixing math chain-of-thought and instruction-constraint data into the fine-tuning set preserves the capabilities that generic conversational SFT erodes.

Data mix

Conversations longer than 2,048 tokens were dropped.

Training details

• Method: QLoRA (4-bit NF4 base, LoRA r=16, alpha=32, dropout 0.05) on all attention and MLP projections of the language model; the vision tower is untouched. Merged to bf16 after training.
• Schedule: 1 epoch, effective batch size 16, lr 1e-4 cosine with 25 warmup steps, max seq len 2048.
• Stack: Unsloth + TRL + PEFT + bitsandbytes on a single RTX 5090 (WSL2). Final training loss ~1.03.

Benchmarks

All models evaluated with lm-evaluation-harness 0.4.13, bf16, chat template applied (--apply_chat_template --fewshot_as_multiturn), on a single RTX 5090. Because these are 0-shot chat-template runs, absolute numbers are not comparable to Google's published figures; only the relative comparison between these four models is meaningful.

Honest read: this model is the strongest of my three fine-tunes, beating the FineTome runs on math, instruction-following and MMLU. Against the base model it gains ~4.6 points on ARC-Challenge and ~1.6 on MMLU, but still loses ~10.6 points on GSM8K and ~9 on IFEval. Mixing in targeted math and constraint data recovered roughly a third of the math regression that generic SFT causes, but did not eliminate it. If you need maximum math or instruction-following accuracy, use google/gemma-3-4b-it instead.

Usage

from transformers import AutoModelForImageTextToText, AutoTokenizer
import torch

model_id = "Jkatdare/gemma-3-4b-it-tulumix-30k"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
    model_id, dtype=torch.bfloat16, device_map="auto"
)

messages = [{"role": "user", "content": "Explain LoRA fine-tuning in one paragraph."}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)
out = model.generate(**inputs, max_new_tokens=300)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

License

Gemma is provided under and subject to the Gemma Terms of Use. This model is a derivative work of Gemma 3 and the same terms apply.