Nawah-VL-50M-detect-ar-stage3

Nawah-VL-50M-detect-ar-stage3

Arabic dense object detection in a 145M-parameter vision-language model. An image goes in; every object it finds comes out, each named in Arabic with a bounding box. There is no prompt to write — the question is a constant.

input   an image
output  منزل 10 00 98 66
        نافذة 17 33 26 42
        نافذة 43 11 55 24
        شجرة 80 00 99 64

Coordinates are x1 y1 x2 y2 in 0–99 integer bins normalised to the image — 4 tokens per box against 16 for floats, which matters at this size. Objects are emitted largest-area first.

This is a detector, not a referring model. Earlier checkpoints in this lineage answer "where is the X I named?"; this one answers "what is here, and where?", which needs the label as well as the box and so cannot be scored by geometry alone.

145.0M total, counted from the checkpoint's own tensor shapes.

Usage

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from PIL import Image

REPO = "oddadmix/Nawah-VL-50M-detect-ar-stage3"
model = AutoModelForImageTextToText.from_pretrained(REPO, dtype=torch.float32).eval()
proc = AutoProcessor.from_pretrained(REPO)

image = Image.open("photo.jpg").convert("RGB")
prompt = (f"{proc.tokenizer.bos_token}<image>"
          "ما الأجسام الموجودة في الصورة؟\n"
          "اذكر كل جسم مع إطاره المحيط.\n")

inputs = proc(text=[prompt], images=[[image]], return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(proc.batch_decode(out[:, inputs["input_ids"].shape[1]:],
                        skip_special_tokens=True)[0])

The prompt is not a suggestion. Those two Arabic strings are the exact training strings. This lineage was initialised from a referring checkpoint and tells the two tasks apart from the prompt alone — paraphrase it and you get referring-style output: one bare box, no name.

Feed it original image bytes where you can. The lineage is measurably sensitive to re-encoding.

Results

Measured on 2,000 held-out rows. The four source shards they come from were excluded from the training corpus before the build started, so nothing here was seen in training.

Read the second column first. [email protected] counts a hit only when the Arabic name matches and IoU ≥ 0.5. The image-blind floor — the 4 commonest labels (وجه, رجل, ملابس, امرأة) emitted at their mean box for every image, no pixels consulted — scores 0.010. Unlike referring grounding, where a fixed full-frame box already scores 0.262, there is almost nothing to win here by not looking, so this headline is close to all real signal.

label_f1 is the exception: naming alone has a 0.129 floor, because a handful of labels are very common. Quote the floor beside it or the number reads better than it is.

The zero-shot column is the referring checkpoint this lineage started from, measured on this task. It scores 0.0 across the board because it answers in the referring format — a bare 00 00 99 99 with no name — so format_ok is 0. Its geometry was already good; the naming behaviour and the output format are what this run bought.

By object size

Recall at IoU 0.5, bucketed by how much of the frame the gold object fills:

Small objects are the known weak band, and attacking it is why this run unfroze the top vision blocks. Recall rose most in exactly the two bands that motivated it, and slipped slightly in the medium band, so the tower reallocated spatial capacity rather than adding it uniformly.

By how crowded the scene is

Recomputed from the 200-prediction dump in results/eval_detection_samples.json — a subset of the 2,000, so treat these as indicative rather than as headline numbers:

Crowding and object size are not independent — a picture with nine things in it has small things in it — but the drop is steep and it is the single most useful thing to know before pointing this at an image.

Counting and repetition

Emitting the same (name, box) twice is this model's characteristic failure — it learns the line format long before it learns to stop. Training targets can never contain a duplicate, so every one is an error the model could filter itself. The dedup row is what it would score with a one-line post-filter.

Training

This is stage 3 of a detection run, on top of a chain of earlier Nawah-VL-50M finetunes: captioning → VQA → referring grounding → detection stage 2 (frozen vision tower) → this. The intermediate checkpoints are internal and are not published; the two published ancestors are oddadmix/Nawah-VL-50M and, under it, oddadmix/50M-2048-Emhotob.

Targets were built by filtering the YOLO detections at confidence ≥ 0.40, capping at 10 objects per image, sorting the survivors largest-area-first, and dropping rows where nothing survived — 23.7% of them. Images were capped at 512 px on the long side and rows with an aspect ratio worse than 5:1 were dropped.

Against stage 2 on the identical pinned rows: [email protected] 0.4722 → 0.4870, label_f1 0.6656 → 0.6694, mean_iou_matched 0.826 → 0.830, eval loss 1.4113 → 1.3713. Recall by band moved tiny 0.273 → 0.294 and small 0.523 → 0.553, while medium slipped 0.723 → 0.711. The margin is modest and has not been tested for significance, and the eval loss was flat over the last 277 updates as the cosine decayed, which is not the same as convergence.

Limitations

The labels are model-generated. The supervision comes from yolov8x-oiv7, a detector, not from human annotation. This model inherits that detector's blind spots, its closed 601-class Open Images vocabulary, and any artefact in the Arabic glossary those class names were mapped through. It cannot name an object that vocabulary has no word for, and "the gold is wrong" is a real explanation for a share of its misses.

Crowded scenes are where it falls apart. Recall roughly halves between a single-object image and one with seven or more, on the dump measured above. Small objects are the other weak band, at 0.294 recall for the tiny quartile against 0.770 for large.

It repeats itself. 8.2% of output lines duplicate an earlier one. A three-line post-filter recovers about 2 points of F1.

It is rigid about the prompt. Anything other than the two exact training strings gets referring-format output, not detection.

It does not read text in images, and inherits the coarse Arabic of a 51.8M-parameter language model. Boxes are quantised to a 100×100 grid, so sub-1% localisation is not representable at all.

Everything reported above is measured on held-out rows from the same corpus the model trained on. There is no evaluation on an independent benchmark, no human-annotated test set, and no COCO-mAP number — this card deliberately does not quote one, because none was computed.

Reproducing

code/ in this repo carries the training loop, the evaluator and the corpus builder that produced these numbers; results/ carries the raw eval JSON, the 200-prediction dump against gold, and the floor and zero-shot baselines. The corpus is public, so the build is repeatable end to end.

Attribution

The language model is oddadmix/50M-2048-Emhotob, trained from scratch on Arabic. The vision tower is SigLIP2 as distributed in LiquidAI/LFM2.5-VL-450M, used unmodified except for its top 4 blocks; those weights remain LiquidAI's and their licence terms apply to that component. Box supervision comes from Ultralytics yolov8x-oiv7 over the Open Images V7 vocabulary, via oddadmix/laion-coco-5m-arabic-labels, which is itself derived from CaptionEmporium/laion-coco-13m-molmo-d-7b and LAION.

License

cc-by-nc-4.0 — non-commercial use only. The detection corpus is CC-BY-SA-4.0, but these weights descend from oddadmix/Nawah-VL-50M, whose captioning data (visheratin/laion-coco-nllb) is CC-BY-NC-4.0, so the non-commercial restriction carries through the initialisation and is the binding term.

Demo

oddadmix/Nawah-VL-50M-Detection-Demo — drop an image in, get boxes with Arabic labels. Six held-out examples included, one of them a crowded scene picked to show the failure above.