Jev-Omni-MLX-4bit

Jev-Omni-MLX-4bit

An Apple Silicon MLX conversion of akhilaaa3/Jev-Omni for local inference on a Mac mini with 16GB unified memory.

This repository keeps the upstream Jev-Omni unified multimodal checkpoint and its trained 256-way decision head, then converts the language-model weights to 4-bit affine quantization with group size 64. The vision weights remain BF16 and the decision head remains FP32.

It is an independent conversion. It is not an official TypeSafe Jev release and does not claim to reproduce TypeSafe's proprietary system. It is also not a new fine-tune.

Hardware and speed

Measured on:

• Mac mini, Apple M4, 10 CPU cores, 16GB unified memory
• macOS 26.5.1
• Python 3.13.12
• MLX 0.32.2
• MLX-VLM 0.7.1
• single request, batch size 1, no token generation

Ten warm requests were measured after one warm-up request. The image test used the recommended 20 visual-token budget and a 3-option question.

The first request includes MLX graph and memory warm-up. On the same machine, a 70-token image request is slower (roughly 1.8 seconds warm in an earlier run). Lowering visual tokens reduces latency but can lose small details; validate on your own game frames.

The published Jev-Omni H200 numbers are not transferable to this Mac mini. This model card reports local measurements only.

Results

The public JevBench files were evaluated with the same typed-choice mapping used by the runtime. Temperature scaling changes probabilities only; it does not change the selected option.

The JevBench result is our local public-set measurement, not a claim that the 4-bit MLX conversion reproduces the upstream card's protocol. The upstream Jev-Omni card reports its own merged-model result separately. Dataset revisions, item filtering and scoring splits must match before comparing the numbers.

Validation

• Upstream unified verification cases: 4/4 argmax decisions matched after 4-bit conversion.
• Six simple red/blue/green circle and square image checks: 6/6 color decisions matched.
• Maximum absolute probability difference on the four upstream text cases: 0.244 in this small check.
• Public JevBench v1.2 public items (195 groups / 231 decisions) were re-run locally. Raw micro accuracy was 87.88% and group-macro accuracy was 85.90%. Raw ECE-10 was 0.04497.
• A single global temperature was fit on even source rows (116 items) and checked on odd rows (115 items): T=1.11517. On all 231 items, ECE-10 was 0.03069 after scaling; the held-out ECE was 0.06774 versus raw 0.06261, so this is a published post-hoc calibration artifact, not a universal confidence guarantee.
• DecisionBench medium was started with the original full states; an exact checkpoint (68/293 questions, 0 errors) is under artifacts/benchmarks/runs/ in the source project. The partial aggregate is deliberately not presented as the official complete score because long states take tens of seconds to minutes on a 16GB Mac.
• The runtime supports the published temperature file through --calibration calibration.json. Accuracy/argmax is unchanged by temperature scaling; only the returned probability distribution changes.

Installation

This release is intended for Apple Silicon. Download the repository and install the small MLX runtime:

hf download Ruiruiz30/Jev-Omni-MLX-4bit \
  --local-dir Jev-Omni-MLX-4bit

cd Jev-Omni-MLX-4bit
python3.13 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Image decision

python -m omni_mlx.classifier \
  --model . \
  --calibration calibration.json \
  --image /path/to/frame.png \
  --state "A kart is approaching a right turn." \
  --question "Which steering action is best?" \
  --options "Turn left" "Hold center" "Turn right" \
  --image-tokens 20

The classifier returns candidate probabilities and the selected option. It does not generate a free-form explanation. Use --image-tokens 70 when the scene contains small or dense visual details.

Omit --calibration to inspect the raw quantized probabilities. The included calibration file was fit only on the public JevBench split described above; it is not trained on a user's game or on private benchmark items.

Benchmark artifacts

• benchmarks/jevbench-4bit-report.json contains the raw and temperature-scaled aggregate metrics.
• calibration.json is the small runtime file consumed by --calibration.
• The benchmark runner and raw checkpoints remain in the source project so the long DecisionBench run can be resumed without putting the full benchmark text into this model repository.

Calibration

Temperature scaling was fit on 116 even-indexed public JevBench rows and checked on 115 odd-indexed rows. The fitted temperature is T=1.11516790625. On the full public set, ECE-10 moves from 0.04497 to 0.03069; on the held-out split it moves from 0.06261 to 0.06774. This is a transparent post-hoc calibration file, not a guarantee of calibrated confidence on game footage.

Local conversion code

omni_mlx/convert.py contains the conversion path used for this release. The original unquantized checkpoint is not bundled here; it can be obtained from the upstream repository under its own license and terms.

License and attribution

Apache-2.0. See LICENSE and NOTICE.md. The upstream model card, Gemma 4 terms, and dataset rights remain authoritative for their respective components.