imajev-2b

<p align="center"> <img src="assets/logo-light.png" class="block dark:hidden" width="300" alt="imajev"> <img src="assets/logo-dark.png" class="hidden dark:block" width="300" alt="imajev"> </p> <h3 align="center">Decisions for real-world cases.</h3> <p align="center">Small open models that read the photos, records and text a business already has and answer in the options you set, with a probability on each and an explicit <i>can't tell</i>. Your system acts when it is sure and hands the rest to a person.</p> <p align="center"> <a href="https://huggingface.co/spaces/mohit67890/imajev"><img alt="Try the live demo" src="https://huggingface.co/datasets/huggingface/badges/resolve/main/open-in-hf-spaces-md.svg"></a> <a href="https://github.com/mohit67890/imajev"><img alt="GitHub" src="https://img.shields.io/badge/GitHub-mohit67890%2Fimajev-111111?style=flat-square&logo=github"></a> <a href="https://mohit67890.github.io/imajev/"><img alt="Website" src="https://img.shields.io/badge/website-checked%20examples-555555?style=flat-square"></a> <a href="https://huggingface.co/datasets/mohit67890/imajev-bench"><img alt="ImajevBench" src="https://img.shields.io/badge/benchmark-ImajevBench-555555?style=flat-square"></a> <img alt="Apache-2.0" src="https://img.shields.io/badge/licence-Apache--2.0-111111?style=flat-square"> </p> <p align="center"><b>imajev-2b</b> is the smallest, most cautious size of the family · other sizes: <a href="https://huggingface.co/mohit67890/imajev-4b">imajev-4b</a> · <a href="https://huggingface.co/mohit67890/imajev-9b">imajev-9b</a><br> <a href="https://huggingface.co/spaces/mohit67890/imajev"><b>Live demo</b></a> · <a href="https://mohit67890.github.io/imajev/">Website</a> · <a href="https://github.com/mohit67890/imajev">Code and results</a> · <a href="https://mohit67890.github.io/imajev/report/">Technical report</a></p> <p align="center"><a href="https://benchmarkheaven.com/jev-models"><img src="assets/ranks-light.png" class="block dark:hidden" width="820" alt="#1 of 91 on JevBench v1.4.2.2 (scored 27 Sep 2026); #1 of 49 on Image JevBench v0.1.3 (released 28 Sep 2026); #3 of 56 on DecisionBench (eng, v1), 28 Sep 2026."><img src="assets/ranks-dark.png" class="hidden dark:block" width="820" alt="#1 of 91 on JevBench v1.4.2.2; #3 of 56 on DecisionBench"></a></p> <p align="center">_{Ranks of the family's <a href="https://huggingface.co/mohit67890/imajev-4b">imajev-4b</a>. This 2B has no official text-board entry.}</p> <p align="center"><b>Independent results</b> (screenshots of the official leaderboards, 28 Sep 2026)</p> <p align="center"><a href="https://benchmarkheaven.com/jev-models"><img src="assets/proof-jevbench-score.png" width="760" alt="JevBench v1.4.2.2 Composite Score: 1 Imajev-4B 67.4, 2 Plumb-4B 65.8, 3 decider-4b v2 64.1, 4 Jev 1.13.0 63.3"></a></p> <p align="center"><a href="https://benchmarkheaven.com/image-jev-bench"><img src="assets/proof-image-jevbench.png" width="760" alt="Image JevBench v0.1.3 composite: 1 Imajev-4B 76.39, 2 Jev-Omni 73.10, 3 NeoHorse Jev 4B 71.94"></a></p> <p align="center"><a href="https://huggingface.co/spaces/Hanno-Labs/decision-bench-leaderboard"><img src="assets/proof-decisionbench.png" width="760" alt="DecisionBench (eng, v1): 1 bosun-v3.1-1.7b 87.29, 2 bosun-v3.1-0.6b 83.20, 3 imajev-4b 79.65, ahead of glm-5.3-flash, jev-1.13, deepseek-v4.1-flash, gpt-5.6-luna"></a></p> <p align="center">_{The family's imajev-4b: JevBench v1.4.2.2, scored 27 Sep 2026 (<a href="https://benchmarkheaven.com/jev-models">board</a>). Image JevBench v0.1.3, released 28 Sep 2026: #1 of 49, 76.39, ahead of Jev-Omni (<a href="https://benchmarkheaven.com/image-jev-bench">board</a>). DecisionBench (eng, v1): #3 of 56 models, 79.65, ahead of GLM-5.3 Flash (320B), Jev 1.13, DeepSeek V4.1 Flash (552B) and GPT-5.6 Luna; the two above are the benchmark team's own models (<a href="https://huggingface.co/spaces/Hanno-Labs/decision-bench-leaderboard">leaderboard</a>).}</p> <p align="center"><img src="assets/request-listing.png" class="block dark:hidden" alt="imajev-4b checks a listing against its photo: listing.color says red, the photo shows beige shoes; the model names listing.color at 0.999 and the app holds the listing"><img src="assets/request-listing-dark.png" class="hidden dark:block" alt="imajev-4b checks a listing against its photo: listing.color says red, the photo shows beige shoes; the model names listing.color at 0.999 and the app holds the listing"></p>

imajev-2b is the smallest and most cautious size: it automates the fewest decisions, and is right most often when it does.

2026-09-26: the 4B tier moved to a new adapter (phase 3: ImajevBench 83.9%, DecisionBench full suite 79.7%, JevBench hard 72.1% as shipped) and is now the family's best size on ImajevBench; this 2B adapter is unchanged and remains the previous generation. See https://huggingface.co/mohit67890/imajev-4b.

What sets it apart

<p><img src="assets/highlights.png" class="block dark:hidden" alt="Five highlights: a photo read against your record; two photos, one decision; a trained can't tell; open, small and local; Jev's contract, now with images"><img src="assets/highlights-dark.png" class="hidden dark:block" alt="Five highlights: a photo read against your record; two photos, one decision; a trained can't tell; open, small and local; Jev's contract, now with images"></p>

• A photo read against your record. Checks a photo against your own fields and names the one that is wrong. Trained on 72k photo-vs-record and two-photo decisions.
• Two photos, one decision. A reference and a target in the same request: shipped against returned, a known-good part against the one on the line.
• A trained can't tell. Every answer carries a probability for unknown, so the app can stop instead of guessing.
• Open, small and local. Apache-2.0, MLX on a Mac or PyTorch on one GPU; photos and customer data never leave your network.
• Jev's contract, now with images. TypeSafe's Jev request and response (POST /v1/systemone), plus images, unknown_probability and abstained. Jev itself is text-only and hosted; its state limit (32k tokens) is larger than imajev's (32 KB).

One request, every answer typed

The exact script we ran against imajev-4b and its output (rounded, usage shortened); 1.15 s on a Mac Studio (four option orders averaged, calibration file applied). Swap the adapter for this size and the request is unchanged.

import json, requests

URL = "http://127.0.0.1:8765/v1/systemone"

listing = {
    "title": "Men's suede boat shoes",
    "color": "red",
    "product_type": "shoe",
}

questions = {
    "contradicted_field": {
        "type": "choice",
        "instructions":
            "Which field of `listing` does this photo contradict?",
        "criteria": {
            "listing.color": None,
            "listing.product_type": None,
            "none of these": "the photo agrees with every field",
        },
    },
    "color_matches": {
        "type": "noul",
        "instructions":
            "The product in the photo matches `listing.color`.",
    },
    "type_matches": {
        "type": "noul",
        "instructions": "The photo shows the kind of product "
                        "given in `listing.product_type`.",
    },
}

request = {"state": {"listing": listing}, "questions": questions}
with open("listing.jpg", "rb") as photo:
    r = requests.post(URL, files={"image": photo},
                      data={"request": json.dumps(request)})
print(json.dumps(r.json(), indent=2))

<details><summary>Result</summary>

{
  "model": "imajev-4b",
  "answers": {
    "contradicted_field": {
      "type": "choice",
      "choice": "listing.color",
      "probabilities": {
        "listing.color": 0.95,
        "listing.product_type": 0.006,
        "none of these": 0.043
      },
      "confidence": 0.919,
      "unknown_probability": 0.007,
      "abstained": false
    },
    "color_matches": {
      "type": "noul",
      "noul": 0.082,
      "unknown_probability": 0.022,
      "abstained": false
    },
    "type_matches": {
      "type": "noul",
      "noul": 0.989,
      "unknown_probability": 0.004,
      "abstained": false
    }
  },
  "usage": {
    "total_ms": 1152.9,
    "input_tokens": 224
  }
}

</details> <p><img src="assets/request-ticket.png" class="block dark:hidden" alt="A support ticket answered in one text-only request: department, urgency and frustration"><img src="assets/request-ticket-dark.png" class="hidden dark:block" alt="A support ticket answered in one text-only request: department, urgency and frustration"></p>

Automate what is clear, route the rest

imajev-2b on the 279 ImajevBench test questions (photos, records and text; 21 whose honest answer is can't tell), raw probabilities, scored with the benchmark's own rule:

The rest go to a person. The benchmark is built to be hard; measure on a few hundred of your own cases before choosing a threshold. Other sizes at 90%: 2B 38% automated at 95.3% right, 4B 58% at 94.5%, 9B 70% at 91.8%.

How it was made

About a million training decisions across the family, in four stages, for about $676 of rented GPU time for the whole project. The 2B went through all four stages: about 500k human-labelled decisions, about 490k more: 416k on new photo and text sources labelled by our 9B, plus 72k photo-vs-record and two-photo decisions (most labelled by the 9B, 17k by construction), then about 23k hard questions kept only when open-weight teachers agreed, then a soft-target continuation on 39,515 rows carrying Qwen3.6-35B-A3B's full probability distributions (with the strict slice of the Eikos decisions set (caiovicentino1/eikos-decisions, CC-BY-4.0; attribution and per-source licences in docs/eikos-decisions-usage.md) and 5k replayed image decisions). The shipped adapter is the weight-space average of two adapters: the hard-question adapter and that continuation. Every teacher is open-weight; no Jev outputs, paid-API outputs or JevBench items were used.

This adapter

This repository holds the 2B adapter, the latency tier. It is a LoRA (rank 16, alpha 32) on the language layers of Qwen3.5-2B (revision 15852e8c) plus a 255-code decision readout, in PEFT format at the root and in MLX format under mlx/; the weights are the element-wise average (0.5 / 0.5, LoRA matrices and readout) of the hard-question adapter and its soft-target continuation. Code, server and evaluation harness: https://github.com/mohit67890/imajev. Other tiers: https://huggingface.co/mohit67890/imajev-4b (recommended default), https://huggingface.co/mohit67890/imajev-9b (quality).

Which size? The 2B is for tight latency or memory budgets. The 4B is 11 points higher on ImajevBench and 10 points higher on JevBench hard at about 1.5× the latency; start there unless the 2B's footprint is the point.

Technical specification

Training path. One trainer for every stage (PyTorch + PEFT): cross-entropy on the readout logits (soft targets where a record carries a distribution), AdamW with weight decay 0, linear warm-up then cosine decay to 10% of the peak rate, gradient clipping 1.0, seed 0, 4 GPUs. The soft-target stage adds a rationale loss (weight 0.3, at most 192 tokens) and permutes the options of every question.

Data this size saw.

• Initial run: 594,214 records from an earlier image mixture.
• Stage 1: 504,000 decisions from 36 licence-admitted sources (15 text, 21 image), including 4,000 photo-vs-listing contradictions.
• Stage 2, two passes: 616,964 decisions labelled by the 9B, 475,305 kept (a label is kept only when two option orders agree and the top probability is at least 0.6, or unknown at least 0.5); the pairs pass trained on the 71,630 photo-vs-record and two-photo decisions (17,162 labelled by construction). Targets were blended 0.5 : 0.5 with the untuned 2B's own distribution.
• Stage 3, round 1: 14,112 training records: kept teacher questions (9,368 of 13,386 kept on two-answerer agreement) plus the training share of 8,532 human reasoning items from 10 licensed sets.
• Stage 3, round 2: 7,812 training records: 3,598 new (4,852 of 8,097 kept on three-answerer agreement) + 4,214 replayed from round 1.
• Stage 4: 39,515 records: the stage-3 teacher questions relabelled with Qwen3.6-35B-A3B's probability distributions (thinking mode), 9,880 new hard, judge and programmatic questions, the strict slice of the Eikos decisions set (10,570 rows, open-weight teachers only) and 5,000 replayed image decisions.

Compute. $499.07 of rented GPU time on RunPod through stage 3 plus about $177 for stage 4 (8×H100, 6 h 20 min, all three sizes): about $676 for the whole project, every run included.

Full specification: https://github.com/mohit67890/imajev/blob/main/docs/technical-specification.md

Results (2026-09-24; every number reproducible from the code repository's results/)

Our pre-registered test against the untuned base model on ImajevBench (paired cluster sign-flip over 89 evidence clusters): the shipped imajev-2b vs the untuned Qwen3.5-2B, +11.5 points (200 vs 168) [+3.5, +18.9], p = 0.005. The previous imajev-2b (previous adapter) gave +10.0 [+1.8, +17.8], p = 0.019; the earlier imajev-2b the test was registered with gave +2.9 points, p = 0.572. We report all three.

How this version was chosen. We compared the soft-target checkpoint on its own with its average with the previous imajev-2b. The checkpoint alone gains on the hidden ImajevBench split (74.8% vs 70.8%) but loses one JevBench hard item and three photo+record items on the public test split; the average is ahead of the previous version on every tier (JevBench hard 56.8% → 60.4%, ImajevBench 70.3% → 71.7%, hidden split 70.8% → 74.3%) and passes every ship gate. The ImajevBench test split was one of the no-regression gates for that choice, so the 2B's public ImajevBench number is not a pure held-out estimate; the hidden split is.

Older panels measured on an earlier imajev-2b, before the hard-question stage (not re-run): held-out photo sources 54.6%, real two-image pairs 51.4%.

Calibration

calibration.json (schema 1.1) applies one temperature (1.646) to every question type × option-count bucket, fitted by negative log-likelihood on 150 template-generated JevBench-style items (none from JevBench), also used to pick checkpoints. Temperature scaling never changes an answer, only its probability. unknown offsets are 0: bounded offsets were tested and changed no panel by more than 0.1 points.

Checked through the released server (ECE, uncalibrated → with calibration.json); the off-distribution rows were measured on the previous imajev-2b with its own temperature (1.61) and have not been re-run for this version:

Pooled ECE over all 231 public JevBench items in the served configuration: 0.025. An ECE-fit temperature (2.20) lowers the hard-tier ECE to 0.090 but raises the pooled ECE to 0.075, so the NLL fit is shipped.

On photo-only verification the raw probabilities are already calibrated and the temperature over-softens them; if your traffic is mostly photo-against-record checks, serve without --calibration or fit your own temperature on a held-out sample.

How to use

git clone https://github.com/mohit67890/imajev && cd imajev
python3.11 -m venv .venv && . .venv/bin/activate
pip install -e ".[serve,mlx]"            # Apple silicon;  elsewhere: pip install -e ".[serve,torch]"
python scripts/download_model.py         # pinned Qwen3.5-2B
hf download mohit67890/imajev-2b --local-dir adapters/imajev-2b
# Mac (MLX)
PYTHONPATH=src:scripts python scripts/playground/server.py --adapter adapters/imajev-2b/mlx \
  --calibration adapters/imajev-2b/calibration.json --model-name imajev-2b --port 8765
# Linux / CUDA (PyTorch + PEFT)
PYTHONPATH=src:scripts python scripts/playground/server.py --backend torch --adapter adapters/imajev-2b \
  --calibration adapters/imajev-2b/calibration.json --model-name imajev-2b --port 8765

curl -s http://127.0.0.1:8765/v1/systemone \
  -F 'request={"state":{"listing":{"title":"Blue ceramic mug, 350 ml","colour":"blue"}},
               "questions":{"matches":{"type":"noul","instructions":"Does the photo show the listed item?"},
                            "wrong_field":{"type":"choice","instructions":"Which listing field does the photo contradict?",
                                           "criteria":{"title":null,"colour":null,"none":null}}}}' \
  -F [email protected]

The numbers above were measured with --rotations 4 (four option orders averaged: +1.8 hard on JevBench for the 2B, at about 3× the latency).

Training

• Base: Qwen3.5-2B (Apache-2.0). LoRA r16/α32 on every language-model projection including the DeltaNet projections; 255 single-token option codes read at the decision position through a float32 readout head; vision tower frozen.
• Licence-checked decisions and new photo sources: 504,000 human-labelled image and text decisions from licence-verified sources, then 616,964 decisions on new photo and text sources labelled by an earlier version of imajev-9b, 475,305 kept with a keep rule (two option orders must agree; top probability ≥ 0.6 or unknown ≥ 0.5), then 72,000 state-grounded and two-image (reference vs target) decisions (17,162 labelled by construction).
• Hard-question stage: 2 epochs, lr 3e-5, on 17,898 hard typed questions (documents written by Qwen3.6-27B, kept only when two answerers of different families agree) plus licence-verified human reasoning sets.
• Last part of the hard-question stage: 2 epochs, lr 2e-5, on 9,066 rows (new questions kept only when three open-weight answerers agree unanimously, plus 30% replay of the earlier hard questions). Best checkpoint chosen on held-out dev rows.
• Soft-target stage: 2 epochs, lr 2e-5, on 39,515 rows: the hard questions relabelled with Qwen3.6-35B-A3B's full probability distributions (thinking mode), 9,880 new hard, judge and programmatic questions, the strict slice of the Eikos decisions set (10,570 rows, open-weight teachers only) and 5,000 replayed image decisions; soft-target cross-entropy plus a rationale loss (0.3) with option permutation. The shipped adapter is the weight-space average of the previous adapter and this stage's best checkpoint (0.5 / 0.5, LoRA matrices and readout).

Data and licence posture

• Adapter, readout and code: Apache-2.0. Base model Qwen3.5-2B: Apache-2.0.
• No JevBench items (8-gram contamination lint), no outputs from Jev, and no outputs from any paid API were used in training. All teacher models are open-weight.
• Per-source licence receipts are in the code repository. New photo sources: PD12M (CC0), Wikimedia Commons (CC-BY-4.0, CC-BY-3.0 or CC0, checked per file), Open Images (CC-BY-2.0). 16 of the 21 image sources from the first training stage are used under their annotation licences only, with photos under upstream terms (not redistributed); for abo, vizwiz, vizwiz_quality and defects the grant covers the images too.

Limitations

• Single pass, no reasoning at inference: multi-step arithmetic and judging trail larger and reasoning models by a wide margin.
• The last part of the hard-question stage cost 5.6 points on our reasoning dev set, also used for checkpoint selection (64.5% → 58.9%); the soft-target checkpoint recovers to 62.7%, and the shipped average was not measured on that set. MMLU did not move, so the loss looks specific to that set's item styles, but it is a measured regression.
• Abstains too rarely on ImajevBench's Unknown items (5/21 correct, vs 18/21 for the 4B and 15/21 for the 9B).
• Over-confident without calibration.json. Counting is weak. At most two images, 32 KB state, 254 options, 8 questions per request. English only. No free text.

Intended use

Typed decisions inside applications where latency or memory is tight: photo-against-record checks, routing, extraction into fixed option sets, abstention when evidence is missing. Not a safety classifier, not a certificate of correctness, and not for decisions about people without human review.

Citation

@software{imajev2026,
  author = {Garg, Mohit},
  title = {imajev: an open Jev-style typed decision model family for images and text},
  year = {2026},
  url = {https://github.com/mohit67890/imajev}
}

ImajevBench, the photo-and-text benchmark released alongside: https://huggingface.co/datasets/mohit67890/imajev-bench.