JoyAI-VL-Interaction-MLX-4bit

JoyAI-VL-Interaction MLX 4-bit

An MLX 4-bit conversion of jdopensource/JoyAI-VL-Interaction for Apple Silicon.

This is a community conversion, not an official JD release. The original model, project, paper, and training assets belong to their respective authors.

Model details

The nominal weight quantization is 4-bit. The average is higher because some parameters are intentionally left unquantized by the converter.

MLX-VLM usage

pip install -U mlx-vlm

from mlx_vlm import generate, load
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

model_id = "xiaowangzhixiao/JoyAI-VL-Interaction-MLX-4bit"
model, processor = load(model_id)
config = load_config(model_id)

prompt = apply_chat_template(
    processor,
    config,
    "Briefly describe this image.",
    num_images=1,
)
result = generate(
    model,
    processor,
    prompt,
    image="image.jpg",
    max_tokens=128,
)
print(result.text)

vLLM Metal usage

This conversion was also validated with the experimental vllm-metal backend:

VLLM_PLUGINS=metal \
VLLM_METAL_MULTIMODAL_MODE=multimodal-native \
VLLM_METAL_MEMORY_FRACTION=0.55 \
vllm serve xiaowangzhixiao/JoyAI-VL-Interaction-MLX-4bit \
  --served-model-name JoyAI-VL-Interaction-MLX-4bit \
  --host 127.0.0.1 \
  --port 7060 \
  --max-model-len 8192 \
  --max-num-seqs 1 \
  --limit-mm-per-prompt '{"image":32}'

vllm-metal is experimental and is not part of the upstream JoyAI supported deployment matrix.

Conversion

The conversion was created with:

HF_HUB_OFFLINE=1 python -m mlx_vlm convert \
  --hf-path models/joyai-latest-official \
  --mlx-path models/joyai-latest-mlx-4bit \
  --quantize \
  --dtype bfloat16 \
  --q-bits 4 \
  --q-group-size 64 \
  --q-mode affine

The converter-generated processor_config.json was removed because it was incompatible with the tested MLX/vLLM Metal processor path. The original preprocessor_config.json and video_preprocessor_config.json are retained.

Validation

Validated locally on:

• MacBook Pro with Apple M3 Pro
• 36 GB unified memory
• mlx==0.32.0
• mlx-vlm==0.6.4
• vllm==0.25.1
• vllm-metal==0.3.0.dev20260716042225

Validation results:

• MLX weights loaded successfully on Metal in approximately 1.5 seconds.
• vLLM Metal completed KV-cache and multimodal warm-up successfully.
• A real image request completed in approximately 22 seconds.
• The test request used 3,379 prompt tokens and generated 52 completion tokens at approximately 5.2 tokens per second.

These are compatibility checks, not quality benchmarks. Quantization may change output quality relative to the BF16 base model.

Intended use and limitations

• Intended for local research and experimentation on Apple Silicon.
• The model uses frame-by-frame image inputs in the tested streaming setup.
• Context length and image limits depend on the serving backend and available unified memory.
• Users should review the base model card for intended uses, limitations, training information, and safety considerations.

中文说明

这是 jdopensource/JoyAI-VL-Interaction 最新版权重的社区 MLX 4-bit 量化版本，面向 Apple Silicon 本地运行。量化方式为 affine 4-bit，group size 为 64。已经在 M3 Pro、36GB 统一内存环境完成模型加载、Metal 预热和真实图片推理验证。

这不是京东官方发布版本。使用限制、安全说明和训练信息请以原模型卡为准。

Attribution

• Base model: https://huggingface.co/jdopensource/JoyAI-VL-Interaction
• Project: https://github.com/jd-opensource/JoyAI-VL-Interaction
• Paper: https://arxiv.org/abs/2606.14777

The base model and this converted distribution are provided under the Apache License 2.0. See LICENSE for the full text.