Downloads · 30 days
9
29% of all-time downloads
shibuina/drawvla-verify-smolvlm-256m-lora
drawvla-verify-smolvlm-256m-lora is a image-text-to-text model from shibuina. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as apache-2.0.
LoRA adapter for HuggingFaceTB/SmolVLM-256M-Instruct trained as a sketch-prompt validator (DrawVLA-Verify): given a robot-scene image with a drawn circle (which object) and arrow (where), plus a name-free caption such…
Downloads · 30 days
9
29% of all-time downloads
All-time downloads
31
Public
Repo size
23.1 MB
Likes
0
Public
Click a slice to open those files.
.safetensors23.1 MB · 87%
From the Hugging Face model README
LoRA adapter for HuggingFaceTB/SmolVLM-256M-Instruct trained as a sketch-prompt validator (DrawVLA-Verify): given a robot-scene image with a drawn circle (which object) and arrow (where), plus a name-free caption such as "put this there", decide whether the instruction is right (correct, consistent, sufficient) or wrong.
Trained on shibuina/drawvla-prompt-validation (LIBERO-grounded synthetic prompts; 14 corruption modes + grounding configurations). LoRA r=16, alpha=32, dropout 0.05, all attention/MLP projections, 3 epochs, lr 1e-4 cosine, 3% warmup, seed 17.
| split | accuracy | balanced acc | macro-F1 | MCC | AUROC |
|---|---|---|---|---|---|
| synthetic test | 0.885 | 0.830 | 0.844 | 0.691 | 0.943 |
Recall on invalid prompts 0.944, on valid prompts 0.717. A geometry-only classifier on the same test split reaches 0.932 accuracy, so the synthetic benchmark is largely separable from sketch coordinates; treat the human-sketch numbers as the deployment estimate.
The adapter must be scored the way it was trained: the user turn is the image plus
"<VERDICT_INSTRUCTION>\n\nCaption: <caption>", and the verdict is chosen by the label-span log-likelihood of
verdict: right vs verdict: wrong (see training/vlm_training.py and training/train_local.py --evaluate-adapter
in the DrawVLA repository). Prompting it differently (e.g. a system prompt + bare next-token scoring) roughly halves the margins.
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel
base = "HuggingFaceTB/SmolVLM-256M-Instruct"
model = PeftModel.from_pretrained(AutoModelForImageTextToText.from_pretrained(base, dtype="bfloat16"), "shibuina/drawvla-verify-smolvlm-256m-lora")
processor = AutoProcessor.from_pretrained(base)
Files: adapter_config.json, adapter_model.safetensors, chat_template.jinja, processor_config.json, run_config.json, test_generation_metrics.json, tokenizer.json, tokenizer_config.json, train_metrics.json.