Nawah-VL-50M

Nawah-VL-50M

An Arabic image-captioning VLM, assembled from existing parts rather than trained from scratch.

145M total. Generates Modern Standard Arabic captions for a single image.

Usage

from transformers import AutoProcessor, AutoModelForImageTextToText
from PIL import Image

model = AutoModelForImageTextToText.from_pretrained("oddadmix/Nawah-VL-50M")
proc  = AutoProcessor.from_pretrained("oddadmix/Nawah-VL-50M")

inputs = proc(text=["<s><image>"], images=[[Image.open("photo.jpg").convert("RGB")]],
              return_tensors="pt", padding=True, padding_side="left")
out = model.generate(**inputs, max_new_tokens=40)
print(proc.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])

There is no chat template — the prompt is literally BOS followed by one <image> token, which the processor expands into 32–64 placeholders depending on aspect ratio.

Results

Held-out test split of visheratin/laion-coco-nllb:

Grounding gap is eval loss with images rotated by one position minus loss with correct pairings. Near zero would mean the model ignores the image and generates from a language prior; 2.36 says the captions are driven by pixels.

Training

Two stages on 405,526 image/caption pairs (the arb_Arab slice of LAION-COCO-NLLB):

1. Align — projector only, 2000 updates, LR 1e-3
2. Finetune — projector + language model, 8000 updates, LR 2e-4 / 5e-5

The projector's linear_1 is warm-started from LFM2.5-VL (identical 2048×3072 shape); linear_2 is randomly initialised and then rescaled so visual tokens enter the residual stream at the same magnitude as text tokens. Without that calibration the visual tokens were ~36× larger and alignment stalled.

Limitations

51.8M parameters in the language model (~35M non-embedding) is still well below a standard VLM backbone. Against the 25M variant it fixes most Arabic agreement errors and invents far fewer details, but it still reads text inside images poorly and will occasionally state something confidently wrong. The training captions are themselves doubly machine-generated (BLIP English → NLLB Arabic), which caps how much reference-based metrics mean.

Reproducing

Everything needed is in training/ in this repo. The filtered corpus is published at oddadmix/laion-coco-nllb-arabic-filtered, so data_prep.py can be skipped. training/HOWTO.md covers the four non-obvious parts, including the projector scale calibration without which alignment stalls.

Set ARABIC_VLM_LM=oddadmix/50M-2048-Emhotob and run scripts/run_training_50m.sh.

Attribution

The language model is oddadmix/50M-2048-Emhotob, trained from scratch on Arabic. The vision tower is SigLIP2 as distributed in LiquidAI/LFM2.5-VL-450M, used unmodified and frozen; those weights remain LiquidAI's and their licence terms apply to that component.

License

cc-by-nc-4.0, inherited from the training data — non-commercial use only.

Compared with oddadmix/Nawah-VL-25M

Same vision tower, same 405,526 pairs, same schedule -- only the language backbone differs. The grounding gap rising while loss fell is the part worth noting: the extra capacity went into using the image, not into memorising Arabic caption phrasing. A larger model that had simply learned the caption distribution better would show the opposite.

In practice the 50M fixes most of the 25M's Arabic agreement errors and invents far fewer details. Where the 25M writes زجاجة ماء "سوبر" مع مقبض أحمر for a plain red bottle, this one writes زجاجة ماء حمراء مع غطاء أبيض, which is correct on both colour and part.