fireball-designer-agent

fireball-designer-agent v0.1.0

A LoRA adapter for Qwen/Qwen3-VL-8B-Instruct that turns natural-language design requests into tool-call trajectories for Fireball Designer, a design application with a scripted tool surface (3D, SVG, image, video and document workspaces).

Given a request and the current editor context, the model emits the JSON tool calls that carry it out — or declines and asks for clarification when the request is ambiguous.

⚠️ This is a pipeline smoke test, not a production model. It was trained on 14 examples. That is enough to prove the training → publishing → serving chain works end to end; it is nowhere near enough to teach a tool catalog. Please read Limitations before using it for anything.

Intended use

The adapter is built for one narrow job: mapping Fireball Designer requests onto Fireball Designer's own tool schema. It is not a general-purpose assistant, and the tool names it emits are meaningless outside that application.

It is published openly for provenance and reproducibility — every Fireball Designer Agent version is traceable to an exact base model revision, dataset version, and training commit — rather than because it is broadly useful in its current state.

Usage

The adapter targets the language backbone only, so it loads as an ordinary LoRA on top of the unmodified base model. No merge step is required.

vLLM

vllm serve Qwen/Qwen3-VL-8B-Instruct \
  --max-model-len 4096 \
  --enable-lora \
  --lora-modules designer-agent=fireballenterprise/fireball-designer-agent

Then request "model": "designer-agent" against the OpenAI-compatible endpoint to hit the adapter, or "model": "Qwen/Qwen3-VL-8B-Instruct" to hit the unmodified base — useful for A/B comparison against the baseline numbers below.

PEFT

from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoTokenizer

base = "Qwen/Qwen3-VL-8B-Instruct"
model = AutoModelForImageTextToText.from_pretrained(base, dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "fireballenterprise/fireball-designer-agent")
tokenizer = AutoTokenizer.from_pretrained(base)

Note AutoModelForImageTextToText, not AutoModelForCausalLM — Qwen3-VL's architecture is Qwen3VLForConditionalGeneration, which the causal-LM AutoClass does not map.

Prompt format

Training used a single flat instruction block. Matching it at inference time matters — the adapter was supervised on exactly this shape:

You are the Fireball Designer Agent. Given the user's request and the current Designer context, respond with the tool calls needed to complete it.

Context: {"workspace": ..., "current_state": ..., "available_tools": [...]}
Request: <the user's request>
Tool calls:

The model's completion is a JSON array of {"name": ..., "arguments": {...}} objects.

Training

Loss is computed on the completion only; prompt and padding positions are masked out, so the model is supervised purely on the tool calls rather than on reproducing the context it was given.

Only the language backbone is adapted. The vision tower is untouched — every training example is text-only, so adapting the vision encoder would mean training weights that the data never exercises. The base model's image understanding is therefore unchanged by this adapter.

Data

17 hand-written examples split deterministically into 14 train / 3 validation, with 5 further held-out examples used for evaluation. Each is a Fireball Designer request paired with its expected tool-call trajectory.

Evaluation

Scored against 5 held-out examples on exact tool selection, parameter correctness, schema validity, and whether the model correctly asks for clarification on under-specified requests.

What actually improved. On the two examples that call for tool use, the adapter selects the right tools in the right order, every time. The base model selected none correctly and emitted fewer calls than the task required. The adapter has clearly learned the output format and the shape of the task — which is what this run set out to prove.

What got worse, and why. Three of the five evaluation examples should be answered with a clarifying question rather than tool calls. The adapter answers all three with tool calls. Only 1 of the 14 training examples is a clarification case, so it learned the rule "always emit tool calls" — correctly, from the data it was given. This is a dataset gap, not a training bug, and it is the clearest signal from this run about what the next dataset needs.

Why "completion rate" moved the wrong way. Completion requires parameter_accuracy == 1.0, and the adapter's parameters are still often wrong, so it scores 0.00. The base model's 0.40 is largely an artifact: the harness scores unparseable output as "asked for clarification," and the base model's unparseable output happened to coincide with two of the three clarification examples. Read it as a measurement artifact, not as a capability the fine-tune lost.

Read these numbers as a smoke test, not a benchmark. Five examples means each one moves a metric by 20 percentage points, and the evaluation set is drawn from the same tiny hand-written pool as the training data. They indicate whether the pipeline produces a model that responds in the right shape — not how well it would generalize.

Limitations

• Trained on 14 examples. It has not seen enough data to learn the tool catalog. Expect invented parameters and malformed trajectories.
• It will not ask for clarification. With one clarification example in training, it answers ambiguous and under-specified requests with confident tool calls instead of a question. Do not put it in front of destructive operations without a confirmation step of your own.
• Not a general assistant. Outside Fireball Designer's tool schema its output is not meaningful.
• Text-only supervision despite the vision-language base. It has not been taught to ground tool calls in an image.
• Narrow evaluation. Five held-out examples cannot establish real accuracy.
• No safety tuning beyond what the base model already carries.
• Tool schema is not frozen. Fireball Designer's tool surface is still changing; an adapter trained against one version of that catalog will drift as the catalog moves.

Provenance

Every version is reproducible from repository commit + dataset version + training configuration:

License

Apache 2.0, inherited from the base model.