Muse-Glimmer-30B-ToolCall-LoRA

Muse-Glimmer-30B — agentic tool-calling LoRA

A LoRA adapter that makes Muse-Glimmer-30B reliably commit to a tool call.

Why this finetune is needed

Muse-Glimmer-30B is sold as an agentic model, and it is not short of knowledge about the tools you hand it. Probe the stock model on held-out agentic trajectories and it will name the correct function verbatim in its own reasoning — and then keep deliberating past the point where an agent loop needed a call. Here is the stock model, mid-thought, on a task whose gold action is databutton-submit_app_requirements:

"We need to submit app requirements via databutton-submit_app_requirements. Need name, pitch, sp…"

Correct tool, correctly named, never emitted. The generation budget ran out first. That is not a knowledge gap, it is a decisiveness and format gap, and it is the single thing standing between this model and a working agent loop. On 200 held-out examples the stock model produced a well-formed call only 98.3% of the time when one was required.

Supervised finetuning on real trajectories is the direct fix, and it costs 191.7M trainable parameters — 0.64% of the model.

Results

BFCL v3 is the honest test here. Every Toucan trajectory was generated by MiniMax-M2.5, so gains against Toucan gold are partly gains at imitating that model. BFCL was never trained on.

Does it break anything?

A tool-calling finetune damages a general model in one characteristic way: the model starts emitting call syntax when no tools were offered at all. That failure is invisible to every metric above, so it is measured directly — GSM8K prompts with tools=None, counting any <atem:invoke in the output.

Paired significance on the GSM8K delta — McNemar: 32 fixed by the adapter, 4 broken by it, exact two-sided p = 0.0000 -- significant at 0.05.

Usage

from transformers import AutoModelForImageTextToText, AutoTokenizer
from peft import PeftModel

# transformers >= 5.15 is required: `muse_glimmer` is not a registered model
# type before that, and the checkpoint carries no remote code.
base = AutoModelForImageTextToText.from_pretrained(
    "meta-models/Muse-Glimmer-30B", dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(base, "PursuitOfDataScience/Muse-Glimmer-30B-ToolCall-LoRA")
tok = AutoTokenizer.from_pretrained("meta-models/Muse-Glimmer-30B")

msgs = [{"role": "user", "content": "What's the weather in Chicago?"}]
tools = [{"type": "function", "function": {
    "name": "get_weather",
    "description": "Current weather for a city.",
    "parameters": {"type": "object",
                   "properties": {"city": {"type": "string"}},
                   "required": ["city"]}}}]
ids = tok.apply_chat_template(msgs, tools=tools, add_generation_prompt=True,
                              return_tensors="pt").to(model.device)
print(tok.decode(model.generate(ids, max_new_tokens=256)[0, ids.shape[1]:]))

The model answers in Muse-Glimmer's native ATEM syntax, on the to= recipient channel — the adapter does not change the interface, only the reliability of using it.

Training

The perception encoder and vision adapter are frozen. This corpus is text; adapting a ViT on text trajectories damages the multimodal path for nothing.

Data preparation

Three defects in the raw corpus are filtered, because each teaches the opposite of the goal:

• 7.4% of records leak Toucan's own <tool_call>{...}</tool_call> JSON into assistant prose — a third syntax this model's template never uses. Lifted into structured calls.
• 4.2% of gold calls name a tool that was never offered; 114 are literally named unknown. Dropped — supervising those teaches exactly the hallucination the finetune is meant to remove.
• thinking_cot interleaves the agent's planning with [Tool Simulation — …] blocks, which are the data generator inventing tool results. Only the agent passes are kept; an agent must never be trained to hallucinate observations.

Validation is deduplicated against training by normalised opening prompt. Toucan's subsets overlap by construction — single-turn-diversify rewrites the same seeds as single-turn-original — so distinct uuids are not distinct problems, and a naive split leaks 4.6% of validation.

The abstention hard negatives are manufactured from training data at zero labelling cost: take a trajectory, delete the tool it needs, keep the question, and expect an honest refusal. Over-calling is the failure that actually hurts a deployed agent, and Toucan's own irrelevant subset is small.

Provenance and licensing

• Base model Muse-Glimmer-30B, Apache-2.0. Meta also publishes a USAGE_POLICY.md acceptable-use policy which travels with any derivative.
• Training data Toucan-1.5M, Apache-2.0.
• Toucan is synthetic: every trajectory used here was generated by MiniMax-M2.5. The behaviour this adapter installs is distilled from that model's outputs and inherits its habits. BFCL is reported precisely because it is not from that distribution.

Limitations

• Trained for 700 steps on 16,800 examples. Validation loss was still falling when the step budget ran out (0.5038 at step 150 -> 0.4753 at step 700), so this is not a converged finetune.
• The tool-calling evaluation scores the first call of a trajectory. It does not measure multi-turn task completion.
• Muse-Glimmer is multimodal. The vision tower was frozen, but the language tower was adapted and image understanding was not re-measured. Treat the multimodal path as untested after this adapter.
• Single seed. Small differences between arms are not meaningful; the paired test above is reported for exactly that reason.
• Sampling error is roughly ±6.9 points per cell at these n.