FunctionGemma-270M-banking77-router

FunctionGemma 270M BANKING77 Router

This is a full fine-tune of google/functiongemma-270m-it that turns an English banking request into one of ten structured support tool calls. It is a learning experiment, not a production banking system.

What was trained?

BANKING77 is normally a classification dataset with text, integer label, and human-readable label_text fields. This experiment does not add a classification head. It converts label_text into the expected assistant tool call and fine-tunes FunctionGemma to generate that native structure.

{
  "text": "My card is gone. I think it was stolen.",
  "label_text": "lost_or_stolen_card"
}

becomes a target call to handle_lost_or_stolen_card.

Tool schema

Every tool receives the original customer message. One complete schema is:

{
  "type": "function",
  "function": {
    "name": "handle_lost_or_stolen_card",
    "description": "Handle a card reported lost or stolen.",
    "parameters": {
      "type": "object",
      "properties": {
        "customer_message": {
          "type": "string",
          "description": "The original customer support message."
        }
      },
      "required": [
        "customer_message"
      ]
    },
    "return": {
      "type": "string"
    }
  }
}

Use the model

import re

from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "lxyuan/FunctionGemma-270M-banking77-router"
TOOL_DESCRIPTIONS = {
    "handle_card_arrival": "Handle questions about when a newly ordered card will arrive.",
    "handle_card_not_working": "Handle reports that a physical bank card does not work.",
    "handle_cash_withdrawal_not_recognised": "Handle an unrecognized cash withdrawal.",
    "handle_change_pin": "Handle requests to change a card PIN.",
    "handle_compromised_card": "Handle reports that card details may be compromised.",
    "handle_lost_or_stolen_card": "Handle a card reported lost or stolen.",
    "handle_pending_card_payment": "Handle a card payment that is still pending.",
    "handle_terminate_account": "Handle requests to close a bank account.",
    "handle_transfer_not_received_by_recipient": "Handle a transfer the recipient has not received.",
    "handle_verify_my_identity": "Handle questions about completing identity verification."
}


def make_tool(name: str, description: str) -> dict:
    return {
        "type": "function",
        "function": {
            "name": name,
            "description": description,
            "parameters": {
                "type": "object",
                "properties": {
                    "customer_message": {"type": "string"},
                },
                "required": ["customer_message"],
            },
            "return": {"type": "string"},
        },
    }


tools = [make_tool(name, description) for name, description in TOOL_DESCRIPTIONS.items()]
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
message = "My card was stolen last night"
inputs = tokenizer.apply_chat_template(
    [
        {
            "role": "developer",
            "content": "You route customer requests by calling exactly one banking support tool.",
        },
        {"role": "user", "content": message},
    ],
    tools=tools,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)
output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
generated = tokenizer.decode(
    output[0, inputs["input_ids"].shape[1] :],
    skip_special_tokens=False,
)
match = re.search(
    r"<start_function_call>call:([a-z0-9_]+).*?<end_function_call>",
    generated,
    re.DOTALL,
)
if match is None:
    raise ValueError(f"No complete function call: {generated!r}")
print({"tool": match.group(1), "raw_call": match.group(0)})

The model selects a call; it does not execute the tool. Validate arguments and dispatch through an explicit allow-listed handler map.

Observed held-out examples

Input:  I am sick of this damn company and want to close out my account.
Output: <start_function_call>call:handle_terminate_account{customer_message:<escape>I am sick of this damn company and want to close out my account.<escape>}<end_function_call>

Loss function

This run uses TRL SFTTrainer with loss_type="chunked_nll". This is the standard causal-language-model next-token negative log-likelihood, or cross-entropy, computed in memory-saving chunks:

loss = mean(-log P(correct next token | previous tokens))

assistant_only_loss=False, and this is a conversational language-modeling dataset rather than a prompt-completion dataset. Therefore every non-padding token in the rendered developer prompt, tool declarations, user message, and assistant call contributes. Padding labels use -100 and are ignored. Exact generated tool accuracy is a separate application metric and is not the differentiable loss. Chunking does not alter this token selection; the label mask controls which tokens contribute, while loss_type controls how the same calculation is held in memory. See the TRL 1.12 SFT documentation.

Results

Training loss can continue falling while validation loss rises because the model becomes more confident on repeated training rows without improving equally on unseen rows. Cross-entropy can increase from a few confidently wrong tokens even when average token accuracy changes little.

TensorBoard event files, trainer_state.json, and training_metrics.json are included in this repository for inspecting the training curves and recorded results.

Training configuration

Software:

• datasets==5.0.1
• tensorboard==2.20.0
• torch==2.11.0+cu128
• transformers==5.16.1
• trl==1.12.0

Precision and limitations

Load the saved FP32 checkpoint without forcing all weights to FP16. The verified FP32 Hub reload produced a valid function call, while forced pure FP16 on a T4 produced padding-only output. Training used FP16 autocast around FP32 master weights.

• Only ten BANKING77 intents are supported, not all 77.
• There is no out-of-scope or refusal route.
• Argument quality was not scored separately from tool-name selection.
• Ambiguous, adversarial, multilingual, or unrelated requests may route incorrectly.
• Do not use this model for financial decisions without production privacy, safety, monitoring, fallback, and human-review controls.

The BANKING77 mirror describes the dataset as CC BY 4.0. FunctionGemma weights remain subject to the Gemma terms.