LaguQA-Gemma4-E2B

LaguQA-Gemma4-E2B

google/gemma-4-E2B-it with the root variant of the LaguQA-Gemma4-E2B-LoRA adapter merged into its weights, so it loads with plain transformers, vLLM, or any tool that does not read PEFT adapters. It is the main model of the LaguQA thesis. Training details, all seven variants, run-to-run noise, and limitations are documented on the adapter's card; this card covers what is specific to the merged copy.

Part of an undergraduate thesis in Informatics, Universitas Ahmad Dahlan.

Baca dalam bahasa Indonesia

Until 2026-09-27 this repository held the LoRA adapter, and its DOI 10.57967/hf/10277 refers to that adapter release (commit 4b12d26, still reachable in the history). The adapter and all its variants now live in LaguQA-Gemma4-E2B-LoRA; code that loads this name with PeftModel.from_pretrained should switch to that repository.

How it was merged

The merged adapter is the root variant: training file laguqa_train.jsonl v1.5, learning rate 4e-4, 2 epochs, LoRA r 16, seed 1 (adapter sha256 c2f661a1…). PeftModel.merge_and_unload() was run on CPU in bfloat16 (torch 2.13.0, transformers 5.16.1, peft 0.20.0). The parameter count is unchanged at 5,104,297,504, and the vision and audio towers of the base model are kept, so the merged model is a drop-in replacement for google/gemma-4-E2B-it. Details are in asal-gabungan.json.

Merging in bfloat16 rounds each weight once more, so the merged model is not bit-identical to base plus adapter. On the 1,200 LaguQA multiple-choice questions it picks the same option as the original adapter predictions on 1,181 (98.4%), the same agreement seen when the adapter itself is rescored on different hardware. Both answer 677 questions correctly (56.4%).

Results in short

One seed. Across three seeds the same setting averages 53.3 on LaguQA, with a range of 4.7 points; see the adapter card.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "IRedDragonICY/LaguQA-Gemma4-E2B"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16")

pesan = [{"role": "system", "content":
           "Kamu asisten yang menguasai lagu nasional dan lagu daerah "
           "Indonesia, termasuk notasi angka dan notasi ABC-nya."},
         {"role": "user", "content": "Siapa pencipta lagu Gugur Bunga?"}]
teks = tok.apply_chat_template(pesan, tokenize=False, add_generation_prompt=True)
keluar = model.generate(**tok(teks, return_tensors="pt").to(model.device),
                        max_new_tokens=192, do_sample=False)
print(tok.decode(keluar[0], skip_special_tokens=True))

The model answers in Indonesian, so the prompts above are kept in Indonesian.

Licence

Apache-2.0, the licence of google/gemma-4-E2B-it. Training data: CC BY-NC 4.0; per-song rights are listed in HAK-CIPTA.md in the dataset repository.