gemma-4-E2B-it-pi-mono-lora

Gemma 4 E2B-it Pi Mono LoRA

The selected best PEFT LoRA adapter configuration for google/gemma-4-E2B-it on badlogicgames/pi-mono.

Base Model

• google/gemma-4-E2B-it

Dataset

• badlogicgames/pi-mono

Selected Hyperparameters

• Method: SFT LoRA, completion-only loss
• Learning rate: 2e-4
• LoRA rank: 16
• LoRA alpha: 32
• LoRA dropout: 0.05
• Max sequence length: 4096
• Max steps: 200
• Per-device train batch size: 1
• Gradient accumulation steps: 16
• Eval size: 256
• Seed: 42
• Target modules: Gemma 4 language-model attention projections and MLP projections

Selection Evidence

Selected by held-out eval loss from the earlier sweep:

Trackio

• Project: training-agents-sft
• Run name: gemma4-e2b-it-pi-mono-lora-lr2e4-r16-a32-len4k
• Group: pi-mono-sft-sweep

Prior Inspect AI Results

These results were reported from a prior Inspect AI evaluation and are not directly comparable to the standardized local pass@1 protocol unless the exact dataset split, prompt template, decoding parameters, scorer, and code extraction are matched.

Standardized Local Evaluation

Protocol:

• Hardware: Apple Silicon MPS
• Base model: google/gemma-4-E2B-it
• Adapter: Chengshuo0723/gemma-4-E2B-it-pi-mono-lora
• Decoding: greedy, do_sample=False
• Metric: pass@1, one sample per task
• HumanEval: openai_humaneval/test, 164 tasks
• MBPP: google-research-datasets/mbpp, sanitized/test, 257 tasks
• max_new_tokens: 384
• Test timeout: 5 seconds per task

Results:

Delta vs base:

Failure type counts:

Known Evaluation Limitations

• Scores depend strongly on prompt template, split, decoding, pass@1/pass@k definition, code extraction, and scorer implementation.
• Standardized local evaluation in the companion project uses greedy decoding, one generation per task, HumanEval 164 tasks, and MBPP sanitized/test 257 tasks.
• This repo contains a LoRA adapter, not a merged full model.
• The SFT data was trained on coding-agent traces, so HumanEval and MBPP mainly test general single-function Python generation. They do not directly measure tool-use behavior, multi-turn repair, or coding-agent workflow competence.

Agent-specific Held-out Trace Evaluation

This evaluation uses the same badlogicgames/pi-mono conversion path as SFT: same chat template, same deterministic held-out split of 256 validation examples, same max_length=4096 filtering, and completion-only assistant loss.

Loss-only evaluation was run on Hugging Face Jobs with A100 large and non-4-bit inference:

The SFT loss is substantially lower than Base on the held-out Pi traces. It is not numerically identical to the training-time eval_loss=0.5740; this local manual scorer is a separate completion-mask check and may differ from TRL SFTTrainer eval details.

Next-action evaluation used the same 256 held-out examples. Each prompt includes context up to the assistant next action; Base and SFT then generated that next action with greedy decoding and max_new_tokens=512. The Pi tool-call template uses Gemma tool-call DSL, not strict JSON, so schema validity below means valid tool-call DSL with required tool fields.

By action type:

Interpretation: the adapter improves coding-agent trace behavior, especially tool-call formatting, tool selection, and shell/read workflow actions. It does not by itself show improved general single-function Python coding ability under the HumanEval/MBPP local protocol.