JoyAI-VL-Interaction-Preview-AWQ-W4A16

JoyAI-VL-Interaction-Preview AWQ W4A16 (LLM 4bit, Vision BF16)

This repository contains an AWQ W4A16 quantized version of jdopensource/JoyAI-VL-Interaction-Preview.

Original model:

• ModelScope/Hugging Face name: jdopensource/JoyAI-VL-Interaction-Preview
• Architecture: Qwen3VLForConditionalGeneration
• Model type: qwen3_vl
• Project: https://github.com/jd-opensource/JoyAI-VL-Interaction
• Paper: https://arxiv.org/abs/2606.14777

This quantized checkpoint is intended for local vLLM inference experiments. It is not an official release from JD.

Quantization

Quantization method:

• Tool: llmcompressor==0.12.0
• Runtime format: compressed-tensors==0.17.1
• Scheme: W4A16_ASYM
• Weight bits: 4
• Group size: 128
• Targets: language-model Linear layers
• Ignored modules: lm_head, model.visual.*

The visual encoder is kept in bf16 to reduce the risk of visual-understanding degradation. This means the checkpoint is not a fully quantized VLM; only the language-model linear layers are quantized.

Observed local size:

Verified Environment

The checkpoint was tested with:

Python 3.10.20
vllm 0.23.0
llmcompressor 0.12.0
compressed-tensors 0.17.1
transformers 5.10.1
2 x RTX 3080 20GB

vLLM detected:

quantization=compressed-tensors
Using MarlinLinearKernel for CompressedTensorsWNA16

vLLM Usage

Start the quantized model backend:

vllm serve /path/to/JoyAI-VL-Interaction-Preview-AWQ-W4A16 \
  --served-model-name JoyAI-VL-Interaction-Preview-AWQ-W4A16 \
  --host 127.0.0.1 \
  --port 8061 \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.85 \
  --limit-mm-per-prompt '{"image":64,"video":1}'

For lower memory usage, reduce context and multimodal limits:

vllm serve /path/to/JoyAI-VL-Interaction-Preview-AWQ-W4A16 \
  --served-model-name JoyAI-VL-Interaction-Preview-AWQ-W4A16 \
  --host 127.0.0.1 \
  --port 8061 \
  --tensor-parallel-size 2 \
  --max-model-len 4096 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 2048 \
  --enforce-eager \
  --gpu-memory-utilization 0.50 \
  --limit-mm-per-prompt '{"image":1,"video":1}'

vLLM-Omni Real-Time Interaction

This model can be used as the main model backend for the JoyAI real-time interaction server from vLLM-Omni:

PYTHONPATH=/path/to/vllm-omni python -m vllm_omni.experimental.fullduplex.joyvl.serving.server \
  --host 127.0.0.1 \
  --port 8070 \
  --main-backend-url http://127.0.0.1:8061/v1 \
  --main-model JoyAI-VL-Interaction-Preview-AWQ-W4A16 \
  --chunk-frames 16

The interaction server exposes an OpenAI-compatible endpoint:

POST http://127.0.0.1:8070/v1/chat/completions

Chinese Output

The model can output Chinese. For real-time video interaction, set the instruction explicitly, for example:

请始终只用简体中文输出，实时简短描述视频中的重要变化；没有重要变化时保持沉默。

Without an explicit Chinese instruction, the model may output English by default.

Minimal Test

curl http://127.0.0.1:8061/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "JoyAI-VL-Interaction-Preview-AWQ-W4A16",
    "messages": [
      {"role": "user", "content": "用一句中文说明你已经启动成功。"}
    ],
    "max_tokens": 64,
    "temperature": 0
  }'

Expected behavior: the model returns a short Chinese response.

Limitations

• This is an experimental quantized checkpoint.
• The visual encoder is not quantized.
• AWQ calibration used text prompts, so this release prioritizes preserving language-model behavior while keeping the vision tower unchanged.
• Running on 8 GB edge devices such as Jetson Orin Nano Super is not guaranteed. Checkpoint size is not equal to runtime memory usage.
• For real-time video, reduce frame sampling rate and image size according to available throughput.

License

This quantized checkpoint follows the license of the original model release: Apache-2.0.

Please also follow the license and usage terms of the original JoyAI-VL-Interaction project.