orchestrix-actions

Orchestrix

Orchestrix is a fine-tuned FunctionGemma 270M model designed for on-device UI orchestration using function calling.
It converts natural language commands into deterministic UI actions, running fully offline via MediaPipe.

Model Description

This is a fine-tuned version of FunctionGemma 270M Instruct specifically trained for natural language-based UI control through function calling. The model interprets user commands and generates structured function calls to manipulate application UI elements.

Base Model: google/functiongemma-270m-it
Fine-tuning Task: Function calling for UI control
Format: MediaPipe compatible formats (.task, .tflite, .litert)
Deployment: On-device inference via Flutter

Intended Use

Primary Use Cases

• Mobile/Desktop Applications: Natural language UI control in Flutter apps
• On-Device AI: Privacy-focused, offline AI interactions
• Function Calling: Demonstrating LLM tool use capabilities
• Accessibility: Voice-controlled UI modifications

Supported Functions

The model is trained to call the following 6 functions:

Training Details

Training Data

The model was fine-tuned on a custom dataset of 3,300 examples mapping natural language commands to function calls. The dataset was synthetically generated with diverse phrasing patterns to ensure robust understanding.

Training data statistics:

• Total examples: 3,300
• Functions: 6 (change_theme, change_background_color, change_text_color, change_text_size, change_app_title, show_alert)
• Pattern variations: 10-12 unique phrasings per function
• Color options: 16 colors
• Theme options: 11 themes
• Text size options: 17 size variations
• App titles: 200+ predefined titles
• Alert variations: 100+ titles and 150+ messages

Training data format (JSONL):

{
  "user_content": "make the background purple",
  "tool_name": "change_background_color",
  "tool_arguments": "{\"color\": \"purple\"}"
}

Training Procedure

• Base Model: FunctionGemma 270M Instruct
• Fine-tuning Method: Supervised fine-tuning on function calling examples
• Training Examples: 3,300 user command → function call pairs
• Data Generation: Synthetic data with template-based variation
• Optimization: Focused on accurate function name and parameter extraction

Usage

Flutter Integration

import 'package:flutter_gemma/flutter_gemma.dart';

// Initialize Flutter Gemma
await FlutterGemma.initialize();

// Create model instance
final model = await FlutterGemma.getActiveModel(
  maxTokens: 1024,
  preferredBackend: PreferredBackend.cpu,
);

// Define tools (must match training data)
final tools = [
  Tool(
    name: 'change_theme',
    description: 'Change application theme',
    parameters: {
      'type': 'object',
      'properties': {
        'theme': {
          'type': 'string',
          'enum': ['light', 'dark', 'system', 'blue', 'green', 'purple', 
                   'orange', 'red', 'material', 'cupertino', 'custom'],
        },
      },
      'required': ['theme'],
    },
  ),
  Tool(
    name: 'change_background_color',
    description: 'Change the background color of the app',
    parameters: {
      'type': 'object',
      'properties': {
        'color': {
          'type': 'string',
          'enum': ['red', 'green', 'blue', 'yellow', 'purple', 'orange', 
                   'white', 'black', 'pink', 'cyan', 'amber', 'teal', 
                   'indigo', 'lime', 'brown', 'grey'],
        },
      },
      'required': ['color'],
    },
  ),
  Tool(
    name: 'change_text_color',
    description: 'Change the text color in the app',
    parameters: {
      'type': 'object',
      'properties': {
        'color': {
          'type': 'string',
          'enum': ['red', 'green', 'blue', 'yellow', 'purple', 'orange', 
                   'white', 'black', 'pink', 'cyan', 'amber', 'teal', 
                   'indigo', 'lime', 'brown', 'grey'],
        },
      },
      'required': ['color'],
    },
  ),
  Tool(
    name: 'change_text_size',
    description: 'Adjust the text size',
    parameters: {
      'type': 'object',
      'properties': {
        'size': {
          'type': 'string',
          'enum': ['small', 'medium', 'large', 'extra large', 'tiny', 
                   '12', '14', '16', '18', '20', '22', '24', '28', 
                   '32', '36', '40', '48'],
        },
      },
      'required': ['size'],
    },
  ),
  Tool(
    name: 'change_app_title',
    description: 'Change the application title',
    parameters: {
      'type': 'object',
      'properties': {
        'title': {
          'type': 'string',
          'description': 'The new title for the application',
        },
      },
      'required': ['title'],
    },
  ),
  Tool(
    name: 'show_alert',
    description: 'Display an alert or notification',
    parameters: {
      'type': 'object',
      'properties': {
        'title': {
          'type': 'string',
          'description': 'The alert title',
        },
        'message': {
          'type': 'string',
          'description': 'The alert message',
        },
      },
      'required': ['title', 'message'],
    },
  ),
];

// Create chat session
final chat = await model.createChat(
  tools: tools,
  supportsFunctionCalls: true,
);

// Send user message
await chat.sendMessage('Switch to dark mode');

Example Interactions

Example 1: Theme Change

User: "Switch to dark mode"
Model Output: {
  "tool_calls": [{
    "name": "change_theme",
    "arguments": {"theme": "dark"}
  }]
}

Example 2: Background Color

User: "make it purple"
Model Output: {
  "tool_calls": [{
    "name": "change_background_color",
    "arguments": {"color": "purple"}
  }]
}

Example 3: Text Styling

User: "make text large and cyan"
Model Output: {
  "tool_calls": [
    {
      "name": "change_text_size",
      "arguments": {"size": "large"}
    },
    {
      "name": "change_text_color",
      "arguments": {"color": "cyan"}
    }
  ]
}

Example 4: App Title

User: "rename app to Task Manager"
Model Output: {
  "tool_calls": [{
    "name": "change_app_title",
    "arguments": {"title": "Task Manager"}
  }]
}

Example 5: Alert Notification

User: "show alert Success with message File uploaded successfully"
Model Output: {
  "tool_calls": [{
    "name": "show_alert",
    "arguments": {
      "title": "Success",
      "message": "File uploaded successfully"
    }
  }]
}

Limitations

1. Limited Function Set: Only supports 6 predefined functions
2. Parameter Constraints: Colors and themes are restricted to predefined enums
3. Language: English only
4. Context: Optimized for UI control tasks, not general conversation
5. Ambiguity: May struggle with highly ambiguous commands not seen during training
6. Synthetic Training: Trained on template-generated data, may differ from real user phrasing

Bias and Ethical Considerations

• The model is trained on synthetic data and may not represent diverse user populations
• Function calling is deterministic and limited to safe UI operations
• No personal data is collected or transmitted (fully on-device)
• Color names follow Western conventions and may not align with all cultural interpretations

Model Card Authors

Vijay K

Model Card Contact

For questions or issues, please contact: [email protected]

License

This model is released under the Gemma License.

Acknowledgments

• Google DeepMind for the Gemma base model
• MediaPipe for on-device inference capabilities
• Flutter Gemma package maintainers

Repository

Training scripts and dataset generation code available in the project repository:

• generate_data.py: Synthetic dataset generation script
• training_data.jsonl: 3,300 training examples
• train.jsonl: Converted training data for fine-tuning