OmniJev

OmniJev — v1.1

Release date: 2026-09-26. Code · 中文说明 · All metrics · Machine-readable reports

This is the 4B OmniJev decision adapter and heads, requiring Qwen/Qwen3.5-4B separately. It produces typed choice/noul/score probabilities without generating answer text. All three sizes use the Qwen3.5 prefix-branch inference path.

git clone https://github.com/tinnel123666888/OmniJev && cd OmniJev
pip install -r requirements.txt
hf download tinnel123/OmniJev --revision v1.1 --local-dir ckpt
hf download Qwen/Qwen3.5-4B --local-dir base

from mso.infer import MSO1
model = MSO1("ckpt", "base")
answers = model.system_one(
    {"images": ["screen.png"]},
    {"visible": {"type": "noul", "instructions": "A person is visible."}},
)

Use the v1.1 code: it reads all still images as a numbered panel and applies the checkpoint's optional noul bias before temperature scaling. A video state remains a frame mosaic. Missing multi-image files raise an error. MSO_NO_PANELS=1 explicitly restores the historical first-image path for controlled comparisons.

The repository contains LoRA weights, decision and ordinal heads, added-token embeddings, tokenizer and processor files. The full Qwen backbone is not bundled. See release_manifest.json for SHA-256 checksums. Earlier weights remain available through the pre-v1.1-20260926 revision; v1.1 pins this release.

Score correction (2026-09-26): use the per-dataset and question-type audit, including question counts and missing coverage. The old A-OKVQA genmcq aggregate used incorrectly generated yes/no labels and is withdrawn. Existing valid multiple-choice predictions score 56.67% / 68.40% / 77.07% for 0.8B / 2B / 4B (750 questions each); these remain custom project tasks, not official benchmark scores. The old overall macro/micro averages also include the invalid labels and must not be treated as validated overall performance. Historical raw reports are retained as evidence.

Mario's mixed 56.05% is not next-action accuracy (43.52% on 193 questions), and the released replays do not establish closed-loop gameplay. OK-VQA 93.47% is candidate selection with the gold answer supplied, not official open-ended VQA. The old POPE row covers only its random subset; safety covers HaGRID. Base/SFT and decision models were not evaluated on identical samples.

v1.2 remains experimental: experimental action interface. This repository still contains v1.1 weights.

Results

Protocol and limitations

The table reports existing evaluations, not a matched-sample ablation. Base models use candidate-answer probabilities (A1_raw), with at most 1,000 questions per family before dev/test splitting. OmniJev and SFT typically use about 1,500 questions per family. Sampling, option truncation and image inputs differ. The 30-family summary excludes music, for which only base results exist.

The released SFT baseline is 0.8B only. Its 41,951 generations were judged and audited: 97 incorrect explicit answer labels were accepted by the raw LLM judge. Audited accuracy is 48.0894% (equal to the parser), rather than the uncorrected judge's 48.3207%.

v1.1 OmniJev uses numbered multi-image panels. The historical SFT and base evaluations used the first still image only. Multi-image SFT retraining and inference over the newly frozen 272,561-question set are not completed. The 20-step distributed smoke checkpoint is not part of this release.

These are project evaluation families, not uniform official benchmark scores. POPE and LongVideoBench were excluded from training; Mind2Web uses official splits. Other families include custom row-level holdouts; the event-timing family uses videos from the Charades-STA training set. Row-level exclusion does not establish episode-level separation.

The main table contains raw evaluation reports, not a new evaluation of the calibrated serving outputs. Released temperature metadata comes from a separate calibration split. The 4B checkpoint also contains biases.noul=0.0531085661, which moves the yes/no boundary; therefore its raw POPE result must not be restated as a serving-path result.

4B has the highest 30-family macro accuracy, but POPE (84.73% vs 86.42% base) and LVB (56.80% vs 59.02% base) remain lower. The 0.8B and 2B models also have substantial POPE regressions. All rows are retained.

License and attribution

Apache-2.0 for the adapter and project code; the Qwen backbone retains its own license. Developed by Beijing Zhongguancun Academy, the Institute of Automation, Chinese Academy of Sciences, and Zevo.