Qwen3-30B-A3B-Instruct-2507-GPTQ-Int4

Qwen3-30B-A3B-Instruct-2507-GPTQ-Int4

Model Details

This model is a int8 model with group_size 128 of Qwen/Qwen3-30B-A3B-Instruct-2507 generated by vastai modelzoo. Please follow the license of the original model.

vllm Inference

• vllm >= v0.11.0
• VVI >= 26.02

vllm serve vastai-ais/Qwen3-30B-A3B-Instruct-2507-GPTQ-Int4 \
    --max-model-len 131072 \
    --tensor-parallel-size 2 \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 8012 \
    --served-model-name Qwen3-30B-A3B-Instruct-2507-GPTQ-Int4 \
    --enforce-eager \

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d ' {
    "model": "qwen",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Who are you?"}
    ],
    "temperature": 1,
    "max_tokens": 512
  } '