qwen3-vl-4b-brand-tools-controller-lora

Qwen3-VL-4B brand-tools controller (LoRA adapter, nim-host v1)

LoRA adapter fine-tuned on Qwen/Qwen3-VL-4B-Instruct (revision ebb281ec70b05090aa6165b016eac8ec08e71b17) to operate the same two deterministic TypeScript tools as the MiniCPM5 controller: verify_company_website and find_customer_facing_pages.

What this model is

A narrow text tool-use controller. The corpus is text-only; the vision tower does no work and was not loaded or trained (FastModel with text_only=True). It learns the same six behaviors: input-form mapping, tool order, argument discipline, clarification, honest status reporting, and expiry/foreign-ID recovery.

Training data

Synthetic only. Trained on sft/train.jsonl of corpus revision 2026-09-21 (1,032 conversations, 2,856 assistant decisions, 43 base families), generated by executing the real tools against deterministic fixture worlds. Zero live-company or human-reviewed training examples. Fixture success is not real-world accuracy.

Recipe

Config note: this run uses the unsloth-recommended alpha/LR; the repo's earlier SAPPAN recipe (alpha=32, LR 1e-4) was started, killed, and is not the shipped adapter.

Evaluation status

Full 258-episode validation on corpus revision 2026-09-21 (training-runs/reports/nim-validation-2026-09-21.md): 258/258 episodes (scenario macro 1.000, every one of the 43 families 6/6) against the untouched base at 0/258. Serving was the unmerged adapter over the same NF4 base, greedy (temperature 0), thinking off, on the nim host (RTX PRO 6000 Blackwell MIG 4g.96gb). The base model failed every episode, most often on verification arguments (VERIFY_ARGUMENT_expected_company 198, VERIFY_ARGUMENT_jurisdiction 192) and the final-answer schema (185). Paired difference +1.000, standard error clustered by family. Synthetic fixture worlds only: fixture success is not real-world accuracy.

Using this adapter

The adapter is a LoRA delta, meant to be attached to the 4-bit base, not merged:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

base = "Qwen/Qwen3-VL-4B-Instruct"
quant = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                           bnb_4bit_compute_dtype=torch.bfloat16)
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, quantization_config=quant,
                                             device_map={"": 0})
model = PeftModel.from_pretrained(model, "G33-k/qwen3-vl-4b-brand-tools-controller-lora")

Call the two tools with the public schemas, one tool call per assistant turn, greedy and with thinking off. The source repository wraps this into an OpenAI-compatible endpoint (training-runs/serve_openai.py) and scores it with the fixture harness:

(cd minicpm5-eval-kit && node --experimental-strip-types scripts/evaluate.ts \
  --mode model --split validation --base-url http://127.0.0.1:30000/v1 \
  --model qwen3vl-adapter --out ../training-runs/reports/qwen3vl-adapter-validation.json)

Full instructions, including the decode and argument-coercion rules for this model family, are in minicpm5-eval-kit/docs/EVALUATION_GUIDE.md section 7 of github.com/geekhoo/minicpm5-2b-voice.

Serving notes

The adapter is keyed for the text-only decoder (model.layers.N). Load it onto a text-only load of the base, not AutoModelForImageTextToText without key handling (unsloth#3560). Qwen-style <tool_call>{json}</tool_call> output must be parsed with schema-typed coercion. Serve greedy, thinking off, enable_thinking=False at render time. Evaluate the unmerged adapter on the NF4 base it trained against.