InternVL3_5-4B-Instruct-W4A16-BF16Vision

InternVL3_5-4B-Instruct W4A16 BF16-Vision

Ampere-friendly serving build of OpenGVLab/InternVL3_5-4B-Instruct. Text-side linears are compressed-tensors W4A16. The vision tower and multimodal connector stay BF16.

Stock proof

docker run --rm -it \
  --gpus all \
  --ipc=host \
  -p 8001:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  OpenGVLab/InternVL3_5-4B-Instruct \
  --served-model-name InternVL3_5-4B-Instruct-stock \
  --trust-remote-code \
  --dtype bfloat16 \
  --max-model-len 4096

Use

docker run --rm -it \
  --gpus all \
  --ipc=host \
  -p 8001:8000 \
  -v /path/to/InternVL3_5-4B-Instruct-W4A16-BF16Vision:/model \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model /model \
  --served-model-name InternVL3_5-4B-Instruct-W4A16-BF16Vision \
  --trust-remote-code \
  --dtype bfloat16 \
  --quantization compressed-tensors \
  --max-model-len 4096 \
  --limit-mm-per-prompt '{"image":1}' \
  --allowed-local-media-path /

Smoke test

python verify.py --url http://localhost:8001/v1/chat/completions

Notes

• Built for RTX 30xx/40xx Ampere cards.
• Vision stays BF16 on disk and at runtime.
• The repo includes the calibration corpus and local smoke_image.png asset used for verification; the image test uses that file directly.