medgemma-1.5-4b-it-MedCalcCaller

MedGemma-1.5-4B-It — MedCalcCaller

A fine-tuned version of google/medgemma-1.5-4b-it specialized for clinical calculator tool-use. Instead of performing arithmetic, the model extracts clinical parameters from unstructured text and generates structured tool calls to a deterministic calculator backend.

Approach

Standard LLMs fail at medical calculations due to three compounding error sources: entity extraction, formula recall, and arithmetic. This model eliminates the latter two entirely by delegating computation to a symbolic calculator engine (OmniCalc) and training the LLM exclusively on the extraction-and-calling task.

Each training example is a complete multi-turn tool-use trajectory:

system prompt → user (clinical note) → model calls calc_info → tool returns schema
→ model calls execute_calc with extracted values → tool returns result → model responds "Done."

The model learns to:

1. Identify the correct calculator from the clinical context
2. Call calc_info to retrieve the exact input field schema
3. Extract values (with units when needed) from the clinical note
4. Call execute_calc with properly structured arguments

Supported Calculators

55 clinical calculators covering formulae, risk scores, and date calculations:

<details> <summary>Full list (click to expand)</summary> </details>

Training Details

Data

• Source: MedCalc-Bench train_data.csv — 10,496 examples after trajectory transformation (42 skipped due to missing/ambiguous fields)
• Transformation: Each static (note, answer) pair was converted into a multi-turn tool-use trajectory using the OmniCalc calculator backend. The model sees the tool schema, extracts values, and receives the execution result.
• Tool format: LM Studio-compatible [TOOL_REQUEST]...[END_TOOL_REQUEST] markers within the Gemma 3 chat template (<start_of_turn>/<end_of_turn>)

Hyperparameters

Training Notes

• Vision tower and multi-modal projector are frozen (text-only fine-tuning)
• Unsloth's SDPA attention patch was replaced with the original transformers FA2 code path to fix a mask-shape bug with padded batches
• No packing — each example is a separate sequence

Evaluation

Evaluated on the full MedCalc-Bench test set (1,100 instances, 20 per calculator) using greedy decoding through the same multi-turn tool-use loop against the OmniCalc calculator backend.

Overall: 84.6% (931/1,100)

Error Analysis

Calculators at 100% are primarily formula-based (the model only needs to extract 2–5 numeric values). Most remaining errors are clinical reading comprehension failures on complex scoring systems — the model misreads or omits criteria from lengthy clinical notes, not arithmetic errors.

Usage

With the OmniCalc backend (recommended)

This model is designed to be used with the OmniCalc calculator backend included in the training repository. The backend handles all computation, unit conversion, and validation.

# Clone the repo
git clone https://github.com/YOUR_USERNAME/calc-caller
cd calc-caller

# Run evaluation
uv run python eval/eval_local.py --model path/to/this/model

With LM Studio

Load the model in LM Studio and configure the OmniCalc tools. The model's tool-call format ([TOOL_REQUEST]...[END_TOOL_REQUEST]) matches LM Studio's default tool-calling convention for models without native tool support.

Prompt Format

The model expects the Gemma 3 chat template with tool definitions injected into the system prompt:

<bos><start_of_turn>system
You are OmniCalc, a clinical calculator assistant.
[... system prompt with calculator list and rules ...]

You can request calls to available tools with this EXACT format:
[TOOL_REQUEST]{"name": "tool_name", "arguments": {"param1": "value1"}}[END_TOOL_REQUEST]

AVAILABLE TOOLS:
[... tool schemas for calc_info and execute_calc ...]
<end_of_turn>
<start_of_turn>user
[clinical note with calculation request]<end_of_turn>
<start_of_turn>model
[TOOL_REQUEST]{"name": "calc_info", "arguments": {"calc_id": "..."}}[END_TOOL_REQUEST]<end_of_turn>
<start_of_turn>user
[tool response with schema]<end_of_turn>
<start_of_turn>model
[TOOL_REQUEST]{"name": "execute_calc", "arguments": {"calc_id": "...", "variables": {...}}}[END_TOOL_REQUEST]<end_of_turn>

Limitations

• Not for clinical use. This model is a research prototype. It must not be used for real-world patient diagnosis or treatment.
• Extraction errors on complex scores. Scoring systems with many binary criteria (HEART, Caprini, Wells PE) remain challenging — the model may miss or misinterpret criteria from lengthy clinical notes.
• Requires a calculator backend. The model does not perform arithmetic. It must be paired with a compatible calculator engine to produce results.
• English only. Trained exclusively on English clinical notes.

Licensing & Terms

This model is a Model Derivative as defined in the Health AI Developer Foundations (HAI-DEF) Terms of Use.

• Foundational model: Use is subject to the HAI-DEF Terms of Use and the HAI-DEF Prohibited Use Policy.
• Model weights: CC BY 4.0
• Medical disclaimer: This model is not a medical device and is not intended for clinical use. It must not be used in any way that would cause a Health Regulatory Authority to deem Google to be a "manufacturer" of a medical device.

Citation

If you use this model, please cite:

@article{khandekar2024medcalcbench,
  title={MedCalc-Bench: Evaluating Large Language Models for Medical Calculations},
  author={Khandekar, Nikhil and Dey, Sestina and Matero, Matthew and Shrestha, Amanuel and Mahowald, Kyle and Ungar, Lyle},
  journal={arXiv preprint arXiv:2406.12036},
  year={2024}
}

NOTICE

HAI-DEF is provided under and subject to the Health AI Developer Foundations Terms of Use found at https://developers.google.com/health-ai-developer-foundations/terms