Trion_8B-FHIR_R4

Trion-8B-FHIR-R4

A fine-tuned Llama-3.1-8B-Instruct model specialized for multi-turn clinical tool-calling over FHIR R4 APIs. Given a clinical instruction, the model plans and issues a sequence of structured function calls (patient lookup, lab queries, vital sign recording, medication ordering, referrals) and terminates with a final structured answer.

Unlike the source dataset's raw GET / POST / FINISH text-action format, this model was trained to emit calls using Llama 3.1's native JSON tool-calling chat template, making it a drop-in fit for standard transformers / chat_template tool-calling pipelines.

Model Details

Benchmarking

Trion-8B-FHIR-R4 was evaluated across 10 EHR tool-calling task categories using 1,100 synthetic test scenarios. Strong performance (90–100%) was achieved in patient lookup by name and date of birth (Task 1: 99%), age calculation (Task 2: 97%), blood pressure recording (Task 3: 100%), and orthopedic referrals (Task 8: 100%). Moderate performance (40–89%) appeared in magnesium checks and conditional IV orders (Task 5: 45%), latest blood glucose readings (Task 7: 50%), potassium checks with conditional orders and lab requests (Task 9: 51%), and latest magnesium readings (Task 4: 44%). Weak performance was observed in average blood glucose over the previous 24 hours (Task 6: 7%) and HbA1c checks with conditional lab requests (Task 10: 26%). On two additional unseen task combinations, accuracy was 56% for latest potassium readings and 20% for average magnesium readings. Overall, the model achieved 61.9% success on core tasks and 38% on unseen scenarios. It produced valid tool-call formats in all 1,100 episodes, but needs improvement in historical lab-result retrieval, conditional clinical workflows, and generalization to unfamiliar tasks.

What Makes This Different From a Naive Fine-tune

The raw dataset ships every row with a repeated, ~2,000+ token FHIR documentation block as the system prompt (conversation[0]), and tool responses returned as full, unabridged FHIR JSON resources (often 3,000+ tokens for a single observation or bundle). Training directly on this would blow past most sequence budgets and waste the model's attention on boilerplate rather than the clinical reasoning that actually matters.

This model's training pipeline instead:

• Drops the bloated system prompt and replaces it with a compact system message plus the dataset's clean instruction field as the user turn.
• Surgically compresses FHIR JSON tool responses (Bundles, Patient, Observation, MedicationRequest, ServiceRequest resources) down to the clinically relevant fields (e.g. Pat:123 John Doe 1980-01-01, Obs:Magnesium=1.2 mg/dL) instead of the full resource payload — typically ~3,000 tokens compressed to ~50.
• Converts raw GET / POST / FINISH action strings into proper OpenAI/Llama-style tool_calls with structured name + arguments, mapped against a fixed 6-function tool schema (below).
• Applies response-only loss masking, so the model is only trained to predict assistant tool-call tokens — not the system prompt, user instructions, or tool outputs it receives.
• Achieves zero truncation: every converted conversation fits inside the 3072-token budget, so no training example is cut off mid-sequence.

Tool Schema

The model was trained against a fixed set of six callable functions:

Each conversation in training ends with a finish call, and the model was taught to alternate correctly between assistant tool calls and tool/user responses.

Intended Use

• Research and prototyping of clinical LLM agents that interact with FHIR-based EHR systems via tool-calling.
• A starting point for building assistants that chain patient lookup → lab/vital retrieval → order placement workflows.
• Educational reference for how to adapt a small, "flat" tool-calling dataset into an efficient instruction-tuned chat model.

Out-of-Scope / Limitations

• Not validated for clinical use. This model has not been evaluated for safety, accuracy, or reliability in real patient care settings and must not be used to make or support real clinical decisions.
• Trained on only 284 examples across 10 task types — coverage of edge cases, ambiguous instructions, and out-of-schema FHIR resources is limited.
• The six-function tool schema is fixed; the model has not been trained to generalize to arbitrary or unseen tool definitions.
• Tool response compression during training discards most FHIR resource fields — the model has not seen full, uncompressed FHIR payloads and may not handle them robustly at inference time if you skip a similar compression step.

Quick Start

Load with Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Rumiii/Trion_8B-FHIR_R4"

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

Tool-Calling Inference

tools = [
    {
        "name": "search_patient",
        "description": "Search for a patient by name and date of birth to retrieve their MRN.",
        "parameters": {
            "type": "object",
            "properties": {
                "given": {"type": "string", "description": "Patient's given (first) name"},
                "family": {"type": "string", "description": "Patient's family (last) name"},
                "birthdate": {"type": "string", "description": "Date of birth, YYYY-MM-DD"},
            },
            "required": ["given", "family", "birthdate"],
        },
    },
    # ... include get_observation, record_vital_sign, order_medication,
    #     order_service_request, and finish as defined in the Tool Schema above
]

messages = [
    {
        "role": "system",
        "content": "You are a clinical assistant with access to FHIR R4 tools. "
                   "Use them to complete the task, then call finish().",
    },
    {
        "role": "user",
        "content": "Look up patient John Doe, born 1980-01-01, and retrieve his most recent magnesium level.",
    },
]

prompt = tokenizer.apply_chat_template(
    messages, tools=tools, tokenize=False, add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

The model should respond with a tool_calls block invoking search_patient, followed (after you return a tool result) by get_observation, and eventually finish.

Training Data

Nadhari/MedToolCalling — 284 multi-turn clinical tool-calling conversations spanning 10 task types, each requiring the model to plan and execute a sequence of FHIR R4 API calls to satisfy a clinical instruction.

Training Procedure

• Framework: Unsloth FastLanguageModel + Hugging Face Trainer
• Quantization: 4-bit NF4 base weights during training, LoRA adapters merged to fp16 for release
• Epochs: 3, with early stopping (patience 3) on eval loss
• Batch size: 1 per device, gradient accumulation 16 (effective batch size 16)
• Optimizer: adamw_8bit, cosine LR schedule, learning rate 2e-4, weight decay 0.01
• Precision: bf16 where supported, fp16 otherwise
• Loss masking: response-only — loss is computed exclusively on assistant tool-call tokens, with system/user/tool turns masked out
• Train/eval split: 90/10, seed 3407

Citation

If you use this model, please cite the base model and dataset it builds on:

@misc{trion8bfhirr4,
  title  = {Trion-8B-FHIR-R4: A Llama-3.1-8B Tool-Calling Model for FHIR R4 Clinical Workflows},
  author = {Rumi},
  year   = {2026},
  url    = {https://huggingface.co/Rumiii/Trion_8B-FHIR_R4}
}

Base model: meta-llama/Llama-3.1-8B-Instruct Dataset: Nadhari/MedToolCalling