gemma-3-4b-minecraft-biomes-lora

Gemma-3-4B — Minecraft Biome Classifier (LoRA)

A LoRA adapter over google/gemma-3-4b-it that classifies Minecraft screenshots into 13 broad biome categories.

Intended use

Given a single Minecraft screenshot, predict the biome as one word from:

desert, plains, forest, taiga, jungle, swamp, savanna,
tundra, ocean, mountain, beach, river, mesa

Target use: labeling or classifying Minecraft RGB frames from agents (MineRL) or gameplay captures. Not intended for non-Minecraft imagery — the model has only ever seen Minecraft screenshots during training.

Performance

Two honest numbers, different distributions:

The MiDaS number is from v1 (before we added MineRL ocean examples). We did not re-run MiDaS after the v2 retrain. Expect similar out-of-distribution behavior — the model is optimized for the willowc

• MineRL visual style and generalizes imperfectly to datasets rendered differently.

Per-class MineRL audit accuracy (95% Wilson CIs)

River is the weakest class — visually ambiguous with swamp/ocean/beach at 640×360 screenshot resolution.

Training

• Base model: google/gemma-3-4b-it (bf16)
• Method: LoRA (r=16, alpha=32, dropout=0.05, all-linear target)
• Trainable params: 38.5M (0.89% of base model)
• Dataset: willowc/minecraft-biomes (20,101 train / 2,231 val samples across 41 numeric biome folders, mapped to 13 broad categories), augmented with 163 hand-selected MineRL ocean frames to correct an ocean→mountain failure mode.
• Augmentation: horizontal flip, brightness ±20%, contrast ±15%, saturation ±20%. Applied to under-represented classes until each folder had at least 100 samples (capped at 5× the number of originals).
• Hardware: 4× GPU DDP via HuggingFace Accelerate
• Hyperparameters: batch size 4 per GPU (effective 16), lr 1e-4, AdamW (weight decay 0.01), gradient clipping 1.0, 3 epochs, gradient checkpointing enabled
• Training time: ~2h 30m on 4 GPUs

Training history (per-step train loss, epoch-mean train/val loss, per-class accuracy) is in train_history.json; a loss curve plot is at loss_curve.png.

How to use

Prerequisite: the base model google/gemma-3-4b-it is a gated repository on HuggingFace. You must first accept the Gemma license on the base model page and log in with hf auth login before from_pretrained will succeed.

from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import PeftModel
from PIL import Image
import torch

base = "google/gemma-3-4b-it"
adapter = "Wafik20/gemma-3-4b-minecraft-biomes-lora"

processor = AutoProcessor.from_pretrained(base)
model = AutoModelForImageTextToText.from_pretrained(
    base, torch_dtype=torch.bfloat16, device_map="cuda:0",
)
model = PeftModel.from_pretrained(model, adapter)

BIOME_CHOICES = "desert, plains, forest, taiga, jungle, swamp, savanna, " \
                "tundra, ocean, mountain, beach, river, mesa"

prompt = (
    "Identify the Minecraft biome shown in this screenshot. "
    f"Reply with exactly one word chosen from: {BIOME_CHOICES}."
)

image = Image.open("minecraft_screenshot.png").convert("RGB")
messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "text", "text": prompt},
]}]
text = processor.apply_chat_template(messages, tokenize=False,
                                     add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=5, do_sample=False)
generated = out[:, inputs["input_ids"].shape[1]:]
print(processor.batch_decode(generated, skip_special_tokens=True)[0].strip())
# -> "ocean" (for example)

Use the exact prompt above. Training-time prompt wording affects results — paraphrasing can reduce accuracy meaningfully.

Dataset

A pseudo-labeled dataset of 1908 MineRL RGBD frames labeled by this model is available at Wafik20/minecraft-biomes.

Known limitations

• Restricted vocabulary. The model will always produce one of the 13 biome words for any input. It has no way to express "not a Minecraft scene" or "none of the above." Garbage in → confidently garbage out.
• 60×360 resolution bottleneck on visually similar classes. River and swamp are genuinely hard to distinguish at this resolution. Expect ~80–85% per-class accuracy on those regardless of training.
• In-distribution vs. out-of-distribution gap. 92% on the train dataset's held-out split vs. 60% on MiDaS (a different data source). Screenshots from unfamiliar rendering setups (shaders, resource packs, modded lighting) will degrade performance.
• Pseudo-label provenance for the MineRL audit samples. Ocean fix examples were hand-picked by the author, not independently labeled. The 96% audit is on predictions the same author reviewed.

License

This adapter is derived from Google's Gemma model and is therefore subject to the Gemma Terms of Use. Use of this adapter requires accepting those terms, including the prohibited-use policy.

Citation

@misc{gemma_minecraft_biome_lora,
  author = {Wafik},
  title  = {Gemma-3-4B Minecraft Biome Classifier (LoRA)},
  year   = {2026},
  url    = {https://huggingface.co/Wafik20/gemma-3-4b-minecraft-biomes-lora},
}

Files in this repo

• adapter_model.safetensors — LoRA weights (~150 MB)
• adapter_config.json — PEFT config (rank, targets, base model ref)
• biome_labels.json — the 13-class label map and numeric biome ID map
• tokenizer.json, tokenizer_config.json, chat_template.jinja, processor_config.json — processor bundle
• train_history.json — per-step train loss, per-epoch val stats, per-class val accuracy
• loss_curve.png — training loss visualization