Nawah-VL-50M-grounding-ar-2ep

Nawah-VL-50M — Arabic visual grounding (2 epochs)

A 50M-parameter vision-language model that takes an image and an Arabic label and returns a bounding box. Trained on oddadmix/blip3-grounding-1m-arabic.

This is the 2-epoch model. The earlier 3,000-step version is kept separately at oddadmix/Nawah-VL-50M-grounding-ar; both are published because they were measured independently.

Read the headline against the image-blind floor

[email protected] is 0.654 — but a constant box that never looks at the image scores 0.262 on these same rows. 26% of the reference boxes cover more than half the image, so a quarter of any headline number here is free. The honest margin is ~0.39, not 0.65.

n = 2000, held-out shard. The image-blind floor is a constant 00 00 99 99 box; a constant box grid-searched on the test rows themselves only reaches 0.297. The un-finetuned base scores 0.0 because it emits no parseable box at all — that is a format floor, not a spatial one, which is why it is the wrong number to brag against.

Accuracy is mostly a function of object size

From the 200 dumped predictions (identical rows for both models, so this is like-for-like):

Two things worth knowing before you use this:

• Small objects are where it fails. Large objects are essentially solved (0.982); tiny ones are not (0.280). In v1 the two smallest bands sat below the image-blind floor — it was doing scene-level attention, not localisation. They now clear it, but the gap to large objects is still the dominant error.
• Multi-box queries are weak. A label matching several objects should produce boxes joined by ;. v1 never did this (0.000); this model manages 0.176. Still the worst band.

format_ok 0.9835 means the output syntax is essentially solved — the remaining error is spatial, not syntactic.

Usage

from transformers import AutoModelForImageTextToText, AutoProcessor
from PIL import Image
import torch

repo = "oddadmix/Nawah-VL-50M-grounding-ar-2ep"
model = AutoModelForImageTextToText.from_pretrained(repo, dtype=torch.bfloat16).cuda().eval()
processor = AutoProcessor.from_pretrained(repo)

image = Image.open("photo.jpg").convert("RGB")
messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "text", "text": "قطة\nحدد الإطار المحيط."},
]}]
text = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = processor(text=[text], images=[[image]], return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
# -> "28 01 72 99"   (x1 y1 x2 y2 in 0-99 bins; multiple boxes joined by " ; ")

Output format. Coordinates are integers in 0–99 bins, x1 y1 x2 y2, relative to the image. Divide by 99 to get [0,1]. Bins rather than floats because they tokenise to exactly 4 tokens where floats cost 16 — which matters a great deal at 50M parameters. Round-trip error is at most half a bin.

Feed it original image bytes. The model is sensitive to re-encoding: re-saving one 158×150 test frame as JPEG q72 moved a correct prediction (00 00 99 16) to the whole image (00 00 99 99).

Training

Initialised from the v1 grounding checkpoint, itself initialised from the VQA-v3 checkpoint, so the model could already read Arabic against an image before it saw a box.

Targets are all boxes for a label at confidence ≥ 0.40, Arabic labels only, capped at 6 boxes; images capped at 512 px on the long side.

A note on the v1 → 2ep jump. v1's loss went flat over its last 500 steps and that was read as convergence. It was not — it was the cosine schedule decaying to zero. Given a fresh schedule the same model fell straight through that apparent floor, from 2.1950 to 1.8270, and [email protected] went 0.498 → 0.654 with it. Loss and IoU had come apart in v1, so the geometry eval is what settles these questions, not the loss curve.

What to try next

The vision tower has never been unfrozen. Given format_ok 0.9835 against tiny-object [email protected] 0.280, the residual error is spatial, and a frozen tower is the obvious cap on spatial precision.

Contents

• weights at the root (model.safetensors, config, processor, tokenizer)
• results/ — eval_grounding.json, the 200-prediction dump, and contact-sheet.html (a self-contained page of 100 predictions drawn over their images, filterable by outcome and object size; the pictures there are re-encoded display copies, the boxes come from the eval, which ran on originals)
• code/ — training, data, eval and contact-sheet sources plus the runner scripts