gemma-3n-E2B-it-Android-Control-84k

Gemma-3n-E2B-it Android Control LoRA Fine-tuned Model

Model Overview

This model is a fine-tuned version of Google's gemma-3n-E2B-it base model with LoRA adaptation for Android UI control tasks.

Training Data

• Dataset: OfficerChul/Android-Control-84k
• Data Format: Mobile UI screenshots paired with user instructions to perform appropriate actions (click, scroll, input, etc.)

Training Data Format Example

{
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant that can identify what action to perform on mobile UI Screenshot given the user instruction."
    },
    {
      "role": "user",
      "content": "<image>Click on the Recording 2"
    },
    {
      "role": "assistant",
      "content": "{\"action_type\": \"click\", \"x\": 561, \"y\": 535}"
    }
  ],
  "images": ["and_ctrl/out_episode_18557_step_001.png"]
}

Training Method

LoRA fine-tuning performed using LLaMA-Factory framework

1. Training Configuration (gemma3n-e2b-it.yaml)

• Base Model: google/gemma-3n-E2B-it
• Training Method: LoRA (Low-Rank Adaptation)
• LoRA Configuration:
  • Rank: 32
  • Target modules: q_proj, k_proj, v_proj, o_proj
• Training Parameters:
  • Batch size: 4 (gradient accumulation: 48)
  • Learning rate: 2e-5
  • Epochs: 5
  • LR scheduler: Cosine
  • Optimizer: AdamW (fused)
  • Precision: bf16
• Additional Settings:
  • Gradient checkpointing enabled
  • Vision tower, multi-modal projector, and language model all trainable
  • DeepSpeed ZeRO-2 utilized

2. Model Merging (gemma3n-e2b-it_lora_sft_merge.yaml)

Merged trained LoRA adapter with base model:

• Base Model: google/gemma-3n-E2B-it

Supported Action Types

• click: Click on specific coordinates
• long_press: Long press action
• scroll: Scroll (up/down)
• input_text: Text input
• navigate_back: Navigate back
• navigate_home: Navigate to home screen
• open_app: Open application
• wait: Wait action

Usage

The merged model can be directly loaded using the Hugging Face Transformers library.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "OfficerChul/gemma-3n-E2B-it-Android-Control-84k"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    trust_remote_code=True
)

Evaluation Results

License

Follows the license terms of the Google Gemma model.

Notes

• This model was developed for research purposes in mobile UI automation and accessibility enhancement
• Proper validation is required when using in production environments