Nawah-VL-50M-grounding-ar

Nawah-VL-50M — Arabic visual grounding

Image + an Arabic label → a bounding box. A 50M-parameter finetune of oddadmix/Nawah-VL-50M, trained on oddadmix/blip3-grounding-1m-arabic.

Ask it كلب and it returns where the dog is.

Results

Measured on 2,000 held-out rows. Read the second column before the first.

The image-blind floor is what a single fixed box covering the whole image scores on these same rows without seeing anything. 26% of reference boxes cover more than half the image, so a quarter of the headline [email protected] is available for free. The honest read is a ~0.20 margin over not looking, not 0.50.

An oracle-tuned constant box (15 15 90 90, chosen by grid search on the test rows) reaches 0.297 — still the same story.

The un-finetuned base model scores 0.0 on every metric, but that is a format floor, not a spatial one: it answers in VQA style (مبنى.) and emits no parseable box at all. format_ok 0.0 → 0.994 is the clearest thing the finetune bought.

Accuracy depends almost entirely on object size

Share of frames reaching IoU ≥ 0.5, by how much of the image the object fills (n=100 sample):

Every large object is a hit; small objects fail. The two smallest bands sit below the 0.262 image-blind floor — for small objects this model is worse than a fixed box that never looks. It has learned coarse scene-level attention, not localisation.

results/contact-sheet.html is a self-contained page showing 100 predictions drawn over their images, filterable by outcome and object size. Open it in a browser.

Usage

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from PIL import Image

REPO = "oddadmix/Nawah-VL-50M-grounding-ar"
model = AutoModelForImageTextToText.from_pretrained(REPO, dtype=torch.bfloat16).eval()
proc = AutoProcessor.from_pretrained(REPO)

image = Image.open("photo.jpg").convert("RGB")
label = "كلب"                      # the Arabic label you want located
instruction = "حدد الإطار المحيط."  # exactly this string — it was trained on it

text = (f"{proc.tokenizer.bos_token}<image>{label}\n{instruction}\n")
inputs = proc(text=[text], images=[[image]], return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(proc.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
# e.g. ["12 40 63 88"] — x1 y1 x2 y2 in 0–99 bins

Output format

x1 y1 x2 y2 as integers in 0–99 bins, normalised to the image, top-left origin. Several boxes are separated by ;. To get pixels, divide by 99 and multiply by width/height.

Bins rather than floats because they tokenise to exactly 4 tokens where floats cost 16 — decisive at 50M parameters. Round-trip error is ≤ half a bin.

Limitations

• Small objects fail. See the size table. Below ~20% of the frame it is worse than a constant box.
• Multi-box is effectively broken. It emits one box even when the label has several referents; on a 100-frame sample it scored 0/9 on multi-box rows. Multibox recall of 0.4488 is the aggregate version of this.
• Reference boxes are detector output, from BLIP3-GROUNDING-50M, not human annotation. Some disagreements are the detector's fault, and the ceiling is "reproduce the detector", not "be right".
• Arabic labels only. Labels came from a context-free translation of open-vocabulary detector strings, so an ambiguous label can name the wrong sense.
• Trained at --max-image-side 512; very large images are resized. Boxes are normalised, so this does not invalidate a label.
• Sensitive to re-encoding. Re-saving a small image as JPEG q72 was enough to move one verified prediction from 00 00 99 16 to 00 00 99 99 (the whole frame). Feed it original bytes where you can.

Training

Stage 2 (vision tower frozen), initialised from the VQA-v3 checkpoint so the model already read Arabic against an image — only the output format was new.

Eval loss went 2.2037 (2000) → 2.1972 (2250) → 2.1950 (2500) → 2.1950 (2750) → 2.1951 (3000). Flat after 2,500 — the last 500 updates bought nothing.

code/ holds the training and eval code (vqa/train.py, vqa/data.py, vqa/evaluate_grounding.py) and the runner scripts. vqa/evaluate_grounding.py is what produced the numbers above; VQA token-F1 is meaningless for boxes, so it scores geometry instead.

Not included: build_grounding.py, which turned the 1M dataset into these training shards, was lost to a truncated file on the training box. The output format is fully specified above and in the dataset card, so it is reproducible, but the original script is gone.

What to try next

The residual error is spatial, not syntactic — the format is at 0.994 while small-object accuracy is at 0.107. Stage 2 keeps the vision tower frozen, which is exactly what caps spatial precision, so --stage 3 is the indicated lever. More steps at this configuration are not: the loss was flat from step 2,500.