nemo-vision-9b-v2

nemo-vision-9b-v2 projector

A 129 M-parameter projector that gives nvidia/NVIDIA-Nemotron-Nano-9B-v2 vision through a LLaVA-style embedding splice. The LLM weights stay bit-identical to the base checkpoint — the projector is the only trained component.

The base model is a hybrid Mamba-2/Transformer LLM that was width- and depth-pruned with Minitron from the 12B Nemotron (56 layers, hidden 4480, only 4 attention layers). NVIDIA ships an official VL only for the unpruned 12B sibling. This adapter is the measured answer to whether the pruned hybrid still holds enough capacity to absorb a vision modality: it does, through the splice path alone.

Full process, training code, and ablations: github.com/benthecarman/nemo-vision.

Usage

git clone https://github.com/benthecarman/nemo-vision
cd nemo-vision && uv sync
CUDA_VISIBLE_DEVICES=0 uv run python scripts/infer.py \
    --image img.jpg --prompt "Describe this image briefly."

scripts/infer.py pulls this projector from the hub by default. First run downloads the LLM (~18 GB) and the RADIO tower; inference needs ~24 GiB VRAM in bf16.

Pipeline: image @ 512 px → RADIOv2.5-B (frozen) → 32x32 patch grid → pixel shuffle f=2 → 256 tokens of 3072 dims → 2-layer MLP projector → spliced into inputs_embeds at marker id 900. Prompt format [INST] {prompt} <IMG> [/INST]; greedy decode (the repo's manual loop — NemotronH lacks generate(inputs_embeds=)).

Artifacts

Results

Vision — held-out COCO val2017 captions, teacher-forced CE (lower is better), same 200 images and prompt for every row:

The LoRA rows are the rejected arms: every adapter parameter added text damage with no vision gain over the projector-only path.

Text — lm-eval 0.4.12, completion mode, one pass (the LLM is untouched, so these are the base model's own scores):

Card numbers for the base model (NeMo-Skills, reasoning-on) are not comparable to this completion-mode column.

Limitations

• Close-up ambiguity confabulates: on a single zebra grooming itself, this model and the official 12B VL both describe a mother and nursing foal. Treat outputs as plausible, not verified.
• Behind the official 12B VL on every vision instrument; this is a capacity experiment, not a production VL model.
• Single-turn [INST] prompt only; no chat mode, no reasoning-mode integration, no safety tuning beyond the base model's.
• 256 fixed vision tokens; no dynamic tiling.

License

• Projector weights (this repo): MIT. Trained on LLaVA-558K and LLaVA-Instruct-150K (CC-BY-4.0 data; images belong to their owners) and COCO val2017 captions (CC-BY-4.0).
• Use requires the LLM — NVIDIA Open Model License.
• Use requires the RADIO tower — NVIDIA Source Code License, non-commercial. Practical use of this adapter inherits that restriction.