gemma-3-4b-it-vlm-react-screenshot-to-code

gemma-3-4b-it-vlm-react-screenshot-to-code

A LoRA adapter for google/gemma-3-4b-it (Gemma 3 4B IT, vision-language) that turns a screenshot of a UI into React component code.

Give it an image of an interface; it returns a single self-contained React component that reproduces the layout.

👉 Live demo: Reubencf/react-screenshot-to-code

Model details

Only the language model's attention projections are adapted — the vision encoder is untouched, so the adapter changes how the model writes code, not how it sees images.

Note on the base model. The training run recorded its base as togethercomputer/gemma-3-4b-it-VLM, which is not a resolvable Hub repo. The bundled config.json matches google/gemma-3-4b-it exactly (hidden 2560, 34 text layers, SigLIP-27 @ 896px, vocab 262208), so base_model_name_or_path has been corrected to point there. unsloth/gemma-3-4b-it is an ungated mirror of the same model if you'd rather not request access.

Training

Short LoRA run, recorded in trainer_state.json:

This is a light adaptation, not a full fine-tune — 26 steps at batch size 1. Expect it to nudge output format and style toward React rather than to teach new visual reasoning. Treat generations as a starting scaffold to edit, not production code.

Usage

Requires transformers >= 5.10.1 — the adapter keys use the refactored model.language_model.layers.* module layout, and will not match older checkpoints.

import torch
from transformers import AutoProcessor, Gemma3ForConditionalGeneration
from peft import PeftModel
from PIL import Image

BASE = "google/gemma-3-4b-it"          # or "unsloth/gemma-3-4b-it" (ungated mirror)
ADAPTER = "Reubencf/gemma-3-4b-it-vlm-react-screenshot-to-code"

processor = AutoProcessor.from_pretrained(BASE)
model = Gemma3ForConditionalGeneration.from_pretrained(
    BASE, dtype=torch.bfloat16, device_map="cuda"
)
model = PeftModel.from_pretrained(model, ADAPTER).merge_and_unload().eval()

image = Image.open("screenshot.png").convert("RGB")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text", "text":
         "Convert this screenshot into a single self-contained React component. "
         "Use Tailwind CSS classes for styling. Return only code."},
    ],
}]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device, dtype=torch.bfloat16)

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=1536, do_sample=True,
                         temperature=0.7, top_p=0.95)

print(processor.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Prompt format

Standard Gemma 3 chat template (chat_template.jinja is included). Images are inserted as <start_of_image> and expand to 256 soft tokens each at 896×896.

<bos><start_of_turn>user
<start_of_image>Convert this screenshot into a React component.<end_of_turn>
<start_of_turn>model

Limitations

• 26 training steps. Output quality is closer to the base model than to a task-specialised one.
• Single-screenshot, single-component. No routing, state management, or multi-page flows.
• Colours, spacing, and font sizes are approximated from the image — expect to tune them.
• Text in the screenshot is transcribed by the vision encoder and can be misread at small sizes.
• Generated code is unverified. Review it before running, as you would any model output.
• Inherits all limitations and the licence of the Gemma 3 base model.

Licence

Governed by the Gemma Terms of Use. The adapter weights are a derivative of Gemma 3 and are subject to the same terms.

Framework versions

• PEFT 0.15.1
• Transformers 5.10.1