tinytune-smollm2-xlam

tinytune-smollm2-xlam

A LoRA fine-tune of HuggingFaceTB/SmolLM2-135M-Instruct that emits structured tool / function calls, trained on Salesforce/xlam-function-calling-60k.

Ships two artifacts: a standalone merged model at the repo root (plain transformers, no PEFT) and the raw LoRA adapter under adapter/. Merge produced with Unsloth.

⚠️ Checkpoint snapshot (~1.67 epochs, step 5000/9000 (56%)); may not be a fully-trained final run.

Results

Greedy generation over 200 held-out test examples:

exact_match is the metric that matters: one wrong argument fails it while barely denting token accuracy, so a low SFT loss can still mean imperfect calls.

Output format

The assistant replies with one or more <tool_call>{"name": ..., "arguments": {...}}</tool_call> blocks.

Usage — merged model

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("anisiraj/tinytune-smollm2-xlam")
model = AutoModelForCausalLM.from_pretrained("anisiraj/tinytune-smollm2-xlam").eval()

messages = [
    {"role": "system", "content": "You are a function-calling assistant. Available tools:\n"
        '{"name": "get_weather", "arguments": {"city": "string"}}'},
    {"role": "user", "content": "What's the weather in Paris?"},
]
inp = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
out = model.generate(inp, max_new_tokens=128, do_sample=False)
print(tok.decode(out[0][inp.shape[1]:], skip_special_tokens=True))

Usage — LoRA adapter

from transformers import AutoModelForCausalLM
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/SmolLM2-135M-Instruct")
model = PeftModel.from_pretrained(base, "anisiraj/tinytune-smollm2-xlam", subfolder="adapter")

Training

LoRA (full-precision) via TRL SFTTrainer: r=16, alpha=32, dropout=0.05, targets = attention + MLP; completion-only (generation) loss; lr 2e-4 cosine, warmup 0.03, batch 4 × grad-accum 4. Custom ChatML+tools template (tools in the system turn).

License & data

Weights derive from HuggingFaceTB/SmolLM2-135M-Instruct (Apache-2.0). Trained on Salesforce/xlam-function-calling-60k — review that dataset's license before downstream use.