JoyAI-VL-Interaction-Preview-AWQ

JoyAI-VL-Interaction-Preview-AWQ

AWQ W4A16 post-training quantization (PTQ) of jdopensource/JoyAI-VL-Interaction-Preview.

This model card focuses on how the quantized checkpoint was produced, not on how to consume the model (for inference usage please refer to the base model card).

Base Model

Quantization Method

Modules Targeted and Excluded

• Targeted: all Linear layers in the text model.
• Excluded from quantization:
  • lm_head
  • All vision components: visual, vision_tower, vision_model, vision_proj, merger

The vision encoder and language model head were intentionally kept in higher precision to preserve visual understanding quality and output embedding fidelity.

The exact excluded module list recorded in config.json is:

["model.visual.blocks.0.attn.qkv", "model.visual.blocks.0.attn.proj", ...]

(fully expanded in the repository's config.json under quantization_config.ignore)

Quantization Recipe (as saved in recipe.yaml)

default_stage:
  default_modifiers:
    AWQModifier:
      mappings:
      - smooth_layer: re:.*input_layernorm$
        balance_layers: ['re:.*q_proj$', 're:.*k_proj$', 're:.*v_proj$']
      - smooth_layer: re:.*v_proj$
        balance_layers: ['re:.*o_proj$']
      - smooth_layer: re:.*post_attention_layernorm$
        balance_layers: ['re:.*gate_proj$', 're:.*up_proj$']
      - smooth_layer: re:.*up_proj$
        balance_layers: ['re:.*down_proj$']
      duo_scaling: true
      n_grid: 20
    QuantizationModifier:
      targets: [Linear]
      ignore: [lm_head, 're:.*visual.*', 're:.*vision_tower.*', 're:.*vision_model.*',
               're:.*vision_proj.*', 're:.*merger.*']
      scheme: W4A16
      bypass_divisibility_checks: false

llmcompressor internally split the request into an AWQ smoothing step followed by a QuantizationModifier step.

Calibration Dataset

Because the original jdopensource/JoyAI-VL-Interaction dataset only contains annotation JSON files and not the actual video assets, calibration was performed on a publicly available video-question-answering dataset that ships with real .mp4 files.

Preprocessing Pipeline

For each sampled QA pair:

1. Open the referenced .mp4 with OpenCV.
2. Extract the first video frame.
3. Convert the frame to a PIL RGB image.
4. Build a single-turn user message:

   [
     {"type": "image"},
     {"type": "text", "text": "<question text>"}
   ]
   

5. Apply the Qwen3-VL chat template (add_generation_prompt=True).
6. Process through the base model's AutoProcessor with max_length=2048 and truncation=True.

The calibration batch therefore consists of image + text tokenized inputs matching the model's expected multimodal format.

Hardware and Software Environment

Notes on the Runtime

• expandable_segments is not supported by the CUDA allocator on Windows, so this option had no effect and was left as a harmless no-op.
• The quantization completed successfully on a single RTX 5090 without model sharding.

Quantization Run Log

Key observations from the run:

• AWQ executed 37 sequential subgraphs.
• All calibration, smoothing, and scale propagation steps completed without errors.
• Model was saved with save_compressed=True, producing the compressed-tensors checkpoint layout.

Checkpoint Contents and Size

Format Notes

This checkpoint is not in legacy AutoAWQ format (quant_method: "awq"). It uses the compressed-tensors format produced directly by llmcompressor:

"quantization_config": {
  "quant_method": "compressed-tensors",
  "quantization_status": "compressed",
  "format": "pack-quantized",
  ...
}

Compatibility with downstream engines:

• transformers can load it as long as compressed-tensors is installed.
• vLLM support depends on the vLLM version understanding quant_method: "compressed-tensors" for Qwen3-VL; use a recent release and verify before deploying.

Known Limitations and Caveats

• Post-training quantization (PTQ) always introduces some accuracy loss compared to the original BF16 checkpoint. No downstream benchmark evaluation was performed on this specific quantized checkpoint yet.
• Deprecated API: the AWQModifier used in this run is the legacy compatibility shim in llmcompressor. Newer releases recommend replacing it with AWQTransformModifier followed by QuantizationModifier.
• The vision encoder was excluded from quantization, so memory savings come almost entirely from the language-model weights.
• Calibration was done on a general-domain academic video QA dataset, not on the original JoyAI-VL-Interaction videos. If you deploy this model for the exact domain the base model was tuned for, you may want to re-quantize with domain-specific calibration data.

Reproducibility

To reproduce this quantization from the base model:

1. Download or clone MBZUAI/VCGBench-Diverse and extract the videos/ directory next to vcgbench_diverse_qa.json.
2. Install the same environment (torch==2.11.0+cu128, torchvision==0.26.0+cu128, llmcompressor, transformers, datasets, opencv-python, Pillow).
3. Run the quantization script with random seed 42, 128 samples, max_seq_length=2048.

License

Same license as the base model jdopensource/JoyAI-VL-Interaction-Preview. Please refer to the base model card for the exact license terms.