dots.vlm1.inst

dots.vlm1

<p align="center"> <img src="figures/logo.png" width="300"/> <p> <p align="center"> &nbsp&nbsp🤗 <a href="https://huggingface.co/rednote-hilab/dots.vlm1.inst">Hugging Face</a>&nbsp&nbsp | &nbsp&nbsp 📄 <a href="https://github.com/rednote-hilab/dots.vlm1/blob/main/assets/blog.md">Blog</a> &nbsp&nbsp | &nbsp&nbsp 🔗 <a href="https://github.com/rednote-hilab/dots.vlm1">GitHub</a>&nbsp&nbsp <br> 🖥️ <a href="https://dotsvlm.xiaohongshu.com/">Demo</a>&nbsp&nbsp | &nbsp&nbsp💬 <a href="https://raw.githubusercontent.com/rednote-hilab/dots.vlm1/main/assets/wechat.png">WeChat (微信)</a>&nbsp&nbsp | &nbsp&nbsp📕 <a href="https://www.xiaohongshu.com/user/profile/683ffe42000000001d021a4c">rednote</a>&nbsp&nbsp </p>

Visit our Hugging Face (click links above) or check out our live demo to try dots.vlm1! Enjoy!

1. Introduction

We are excited to introduce dots.vlm1, the first vision-language model in the dots model family. Built upon a 1.2 billion-parameter vision encoder and the DeepSeek V3 large language model (LLM), dots.vlm1 demonstrates strong multimodal understanding and reasoning capabilities.

Model Highlights:

• NaViT Vision Encoder: Trained entirely from scratch rather than fine-tuning an existing vision backbone. It natively supports dynamic resolution and incorporates pure visual supervision in addition to traditional text supervision, thereby enhancing the upper bound of perceptual capacity. Beyond image captioning datasets, a large amount of structured image data was introduced during pretraining to improve the model’s perceptual capabilities—particularly for tasks such as OCR.
• Multimodal Training Data: In addition to conventional approaches, dots.vlm1 leverages a wide range of synthetic data strategies to cover diverse image types (e.g., tables, charts, documents, graphics) and descriptions (e.g., alt text, dense captions, grounding annotations). Furthermore, a strong multimodal model was used to rewrite web page data with interleaved text and images, significantly improving the quality of the training corpus.

Through large-scale pretraining and carefully tuned post-training, dots.vlm1 achieves near state-of-the-art performance in both visual perception and reasoning, setting a new performance ceiling for open-source vision-language models—while still maintaining competitive capabilities in pure-text tasks.

Special thanks to the DeepSeek team for the excellent DeepSeek V3 model.

2. Performance

3. Usage

Environment Setup

You have two options to set up the environment:

Option 1: Using Base Image + Manual Installation

# Use the base SGLang image
docker run -it --gpus all lmsysorg/sglang:v0.4.9.post1-cu126

# Clone and install our custom SGLang branch
# IMPORTANT: Only our specific SGLang version supports dots.vlm1 models
# NOTE: This installation must be done on EVERY node in your cluster
# We have submitted a PR to the main SGLang repository (currently under review):
# https://github.com/sgl-project/sglang/pull/8778
git clone --branch dots.vlm1.v1 https://github.com/rednote-hilab/sglang sglang
pip install -e sglang/python

Option 2: Using Pre-built Image (Recommended)

# Use our pre-built image with dots.vlm1 support
docker run -it --gpus all rednotehilab/dots.vlm1_sglang:v0.4.9.post1-cu126

Multi-Node Deployment

Our model supports distributed deployment across multiple machines. Here's how to set up a 2-node cluster:

Prerequisites:

• Model: rednote-hilab/dots.vlm1.inst
• Node 1 IP: 10.0.0.1 (master node)
• Node 2 IP: 10.0.0.2 (worker node)

Node 1 (Master - rank 0):

# Recommend downloading model locally to avoid timeout during startup
# Use: huggingface-cli download rednote-hilab/dots.vlm1.inst --local-dir ./dots.vlm1.inst
export HF_MODEL_PATH="rednote-hilab/dots.vlm1.inst"  # or local path like ./dots.vlm1.inst
# Get actual IP address: hostname -I | awk '{print $1}' or ip route get 1 | awk '{print $7}'
export MASTER_IP="10.0.0.1"  # Replace with actual master node IP
export API_PORT=15553

python3 -m sglang.launch_server \
    --model-path $HF_MODEL_PATH \
    --tp 16 \
    --dist-init-addr $MASTER_IP:23456 \
    --nnodes 2 \
    --node-rank 0 \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port $API_PORT \
    --context-length 65536 \
    --max-running-requests 64 \
    --disable-radix-cache \
    --mem-fraction-static 0.8 \
    --chunked-prefill-size -1 \
    --chat-template dots-vlm \
    --cuda-graph-max-bs 64 \
    --quantization fp8

Node 2 (Worker - rank 1):

# Use the same variables as defined in Node 1
export HF_MODEL_PATH="rednote-hilab/dots.vlm1.inst"
export MASTER_IP="10.0.0.1"  # Must match Node 1
export API_PORT=15553

python3 -m sglang.launch_server \
    --model-path $HF_MODEL_PATH \
    --tp 16 \
    --dist-init-addr $MASTER_IP:23456 \
    --nnodes 2 \
    --node-rank 1 \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port $API_PORT \
    --context-length 65536 \
    --max-running-requests 64 \
    --disable-radix-cache \
    --mem-fraction-static 0.8 \
    --chunked-prefill-size -1 \
    --chat-template dots-vlm \
    --cuda-graph-max-bs 64 \
    --quantization fp8

Configuration Parameters

Key parameters explanation:

• --tp 16: Tensor parallelism across 16 GPUs per node
• --nnodes 2: Total number of nodes in the cluster
• --node-rank: Node identifier (0 for master, 1+ for workers)
• --context-length 65536: Maximum context length
• --quantization fp8: Use FP8 quantization for efficiency
• --chat-template dots-vlm: Use custom chat template for dots.vlm model

API Usage

Once the servers are launched, you can access the model through OpenAI-compatible API:

# Use the same MASTER_IP and API_PORT as defined above
curl -X POST http://$MASTER_IP:$API_PORT/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "model",
        "messages": [
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "Please briefly describe this image"
                    },
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
                        }
                    }
                ]
            }
        ],
        "temperature": 0.1,
        "top_p": 0.9,
        "max_tokens": 55000
    }'