smollm2-1.7b-xlam-qlora

SmolLM2-1.7B + QLoRA, function calling

QLoRA adapter (NF4, double quantisation, bf16 compute) trained on top of a 4-bit SmolLM2-1.7B base, for function calling: given a user request and a list of tools, it emits the JSON tool calls that answer it.

72.61% exact match on 2,395 decontaminated held-out examples, against 75.11% for the bf16 LoRA twin trained with an otherwise identical recipe. That 2.5-point gap is significant (paired McNemar, 136 vs 76 discordant pairs, p = 5.1e-05) and it bought 38% less training memory — 5.87 GB peak against 9.48 GB.

It also trained faster than the bf16 LoRA on the same A100 (1h36 against 1h45): 4-bit weights mean fewer bytes moved per token, and dequantisation is arithmetic the GPU has to spare.

Pinned revision for every number here: a7cd2365348afefd29ad03fd11ce8d0235d8ae78 ("End of training").

Quick start

Load the base in 4 bits, and do not merge the adapter. It learned to correct a dequantised NF4 model; running it on a bf16 base is a different model, and merging into 4-bit weights requantises the result.

import json, torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

ADAPTER  = "Chloemp/smollm2-1.7b-xlam-qlora"
REVISION = "a7cd2365348afefd29ad03fd11ce8d0235d8ae78"

bnb = BitsAndBytesConfig(
    load_in_4bit=True, bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16,
)

tok = AutoTokenizer.from_pretrained(ADAPTER, revision=REVISION)
base = AutoModelForCausalLM.from_pretrained(
    "HuggingFaceTB/SmolLM2-1.7B", quantization_config=bnb, dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base, ADAPTER, revision=REVISION).eval()

tools = json.dumps([
    {"name": "get_weather",
     "description": "Get the current weather for a location.",
     "parameters": {"city": {"description": "City name", "type": "str"},
                    "unit": {"description": "celsius or fahrenheit", "type": "str", "default": "celsius"}}},
])

messages = [
    {"role": "system", "content": f"You are a function-calling assistant. Available tools:\n{tools}"},
    {"role": "user", "content": "What's the weather in Lyon in fahrenheit?"},
]

inputs = tok.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=512, do_sample=False,
                         pad_token_id=tok.pad_token_id or tok.eos_token_id)

print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
# [{"name": "get_weather", "arguments": {"city": "Lyon", "unit": "fahrenheit"}}]

Requires bitsandbytes and a CUDA GPU. Greedy decoding is what was evaluated; for batched generation set tok.padding_side = "left".

Scores by level

An LLM-judge audit puts accuracy at 82.29% once references that cannot be derived from the request are excluded — computed from the judge's raw flags; the human-review correction below moves this figure down by roughly one point.

Training

QLoRA: NF4 double-quantised base, bf16 compute, LoRA r=16, α=32, dropout 0.05 on all linear layers (18.1M trainable), lr 2e-4, 3% warmup, 1 epoch (3,563 steps), effective batch 16, gradient checkpointing, max length 2048. One A100, 1h36, 5.87 GB peak.

How it was evaluated

2,395 held-out examples from the same dataset, after removing 105 contaminated rows (93 queries appeared on both sides of the split). Greedy decoding, left padding, max_new_tokens=512 (the longest reference in the eval set is 410 tokens), batch size 32, bf16 on one A100. Scores attach to the pinned revision above, not to main.

Scoring is six nested levels, each strictly harder than the last, so a failure is located rather than just counted. Level 6 is exact match after normalising types (5 == "5"), key order and call order.

For reference, Qwen3-235B-A22B-Instruct prompted with the same format block scores 78.33% on a fixed 300-example subset — statistically indistinguishable from the Instruct-based LoRA on those same 300 (80.00%, paired McNemar p = 0.50).

Limitations

• Known tools only. A random split of xLAM means nearly every tool in the eval set also appears in training. These numbers measure reliable calling of a known toolset on new phrasing. On 20 hand-written queries against four tools absent from xLAM, the fine-tunes score 60–70% against 80% for the un-tuned Instruct model: fine-tuning on xLAM costs some generalisation to unseen tools.
• Irrelevance detection is unmeasured. Nothing here tests whether the model abstains when no available tool answers the request. Expect it to call something anyway.
• Single-turn, English, xLAM-style schemas. No multi-turn, no tool results fed back.
• Ground truth is imperfect, and now measured. An LLM-judge audit flagged exact-match failures whose reference cannot be derived from the request (a city, an ID, a ticker appearing nowhere in the prompt). 83 flagged cases were then reviewed by hand, blind to the judge's verdict: its precision is 80.0% on self-consistent flags [67.0, 88.8] but only 27.3% on self-contradictory ones [15.1, 44.2]. Correcting the raw 20.5% by those rates gives 15.0% of exact-match failures [12.7, 16.6] — about one in seven — or 4.2% of all held-out examples. Only flagged cases were reviewed, so that is a floor, not a ceiling: unusable references the judge missed would push it higher. True accuracy is therefore somewhat higher than exact match suggests.

Links

• Code, full method and results: github.com/chloe-mp/smollm2-xlam-function-calling
• Per-example predictions for every condition: Chloemp/xlam-smollm2-eval-results
• Judge verdicts: Chloemp/xlam-smollm2-judge-results

License and attribution

Weights follow the base model's Apache-2.0 licence. Training data is Salesforce/xlam-function-calling-60k, CC-BY-4.0 — attribute Salesforce AI Research if you use anything derived from it.