smollm2-1.7b-xlam-lora

SmolLM2-1.7B + LoRA, function calling

LoRA adapter that turns the SmolLM2-1.7B base model into a function-calling model: given a user request and a list of available tools, it emits the JSON tool calls that answer it.

75.11% exact match on 2,395 decontaminated held-out examples. If you want the strongest adapter in this family, use smollm2-1.7b-instruct-xlam-lora instead (83.47%) — same recipe, started from the Instruct checkpoint. This one is the control that makes that comparison meaningful.

Pinned revision for every number here: 8868bcd45fb4942552222602f539148799d6e593 ("End of training").

Quick start

import json, torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

ADAPTER  = "Chloemp/smollm2-1.7b-xlam-lora"
REVISION = "8868bcd45fb4942552222602f539148799d6e593"

tok = AutoTokenizer.from_pretrained(ADAPTER, revision=REVISION)
base = AutoModelForCausalLM.from_pretrained(
    "HuggingFaceTB/SmolLM2-1.7B", dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base, ADAPTER, revision=REVISION).merge_and_unload().eval()

tools = json.dumps([
    {"name": "get_weather",
     "description": "Get the current weather for a location.",
     "parameters": {"city": {"description": "City name", "type": "str"},
                    "unit": {"description": "celsius or fahrenheit", "type": "str", "default": "celsius"}}},
])

messages = [
    {"role": "system", "content": f"You are a function-calling assistant. Available tools:\n{tools}"},
    {"role": "user", "content": "What's the weather in Lyon in fahrenheit?"},
]

inputs = tok.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=512, do_sample=False,
                         pad_token_id=tok.pad_token_id or tok.eos_token_id)

print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
# [{"name": "get_weather", "arguments": {"city": "Lyon", "unit": "fahrenheit"}}]

The tokenizer shipped here carries the chat template the model was trained with (ChatML, eos <|im_end|>), borrowed from SmolLM2-1.7B-Instruct — the base model has no template of its own. Greedy decoding is what was evaluated; for batched generation set tok.padding_side = "left".

Scores by level

An LLM-judge audit of these predictions puts accuracy at 84.55% once references that cannot be derived from the request are excluded — computed from the judge's raw flags; the human-review correction below shrinks the unusable share and moves this figure down by roughly one point. The judge agreed with 99.89% of the exact matches it was asked to re-grade, which is what makes that figure usable.

Training

LoRA r=16, α=32, dropout 0.05 on all linear layers (18.1M trainable of 1.73B, 1.05%), lr 2e-4, 3% warmup, 1 epoch (3,563 steps), effective batch 16 (8 × 2 accumulation), bf16, gradient checkpointing, max length 2048. One A100, 1h45, 9.48 GB peak.

How it was evaluated

2,395 held-out examples from the same dataset, after removing 105 contaminated rows (93 queries appeared on both sides of the split). Greedy decoding, left padding, max_new_tokens=512 (the longest reference in the eval set is 410 tokens), batch size 32, bf16 on one A100. Scores attach to the pinned revision above, not to main.

Scoring is six nested levels, each strictly harder than the last, so a failure is located rather than just counted. Level 6 is exact match after normalising types (5 == "5"), key order and call order.

For reference, Qwen3-235B-A22B-Instruct prompted with the same format block scores 78.33% on a fixed 300-example subset — statistically indistinguishable from the Instruct-based LoRA on those same 300 (80.00%, paired McNemar p = 0.50).

Limitations

• Known tools only. A random split of xLAM means nearly every tool in the eval set also appears in training. These numbers measure reliable calling of a known toolset on new phrasing. On 20 hand-written queries against four tools absent from xLAM, the fine-tunes score 60–70% against 80% for the un-tuned Instruct model: fine-tuning on xLAM costs some generalisation to unseen tools.
• Irrelevance detection is unmeasured. Nothing here tests whether the model abstains when no available tool answers the request. Expect it to call something anyway.
• Single-turn, English, xLAM-style schemas. No multi-turn, no tool results fed back.
• Ground truth is imperfect, and now measured. An LLM-judge audit flagged exact-match failures whose reference cannot be derived from the request (a city, an ID, a ticker appearing nowhere in the prompt). 83 flagged cases were then reviewed by hand, blind to the judge's verdict: its precision is 80.0% on self-consistent flags [67.0, 88.8] but only 27.3% on self-contradictory ones [15.1, 44.2]. Correcting the raw 20.5% by those rates gives 15.0% of exact-match failures [12.7, 16.6] — about one in seven — or 4.2% of all held-out examples. Only flagged cases were reviewed, so that is a floor, not a ceiling: unusable references the judge missed would push it higher. True accuracy is therefore somewhat higher than exact match suggests.

Links

• Code, full method and results: github.com/chloe-mp/smollm2-xlam-function-calling
• Per-example predictions for every condition: Chloemp/xlam-smollm2-eval-results
• Judge verdicts: Chloemp/xlam-smollm2-judge-results

License and attribution

Weights follow the base model's Apache-2.0 licence. Training data is Salesforce/xlam-function-calling-60k, CC-BY-4.0 — attribute Salesforce AI Research if you use anything derived from it.