qwen3vl-8b-chartqa-awq

Qwen3-VL-8B ChartQA — AWQ W4A16 g32

Deployment-oriented 4-bit checkpoint of a ChartQA-fine-tuned Qwen3-VL-8B-Instruct model. The language Linear weights use symmetric AWQ W4A16 with group size 32; the vision tower and lm_head retain their original precision.

Lineage

1. Base: unsloth/Qwen3-VL-8B-Instruct-unsloth-bnb-4bit
2. QLoRA adapter: steven0226/qwen3vl-8b-chartqa-lora
3. Merged reference: steven0226/qwen3vl-8b-chartqa-merged-16bit
4. This AWQ checkpoint

Formal quantization used an NVIDIA A100-SXM4-40GB, 256 ChartQA calibration examples, maximum calibration length 2,048, seed 3407, llm-compressor 0.12.0, and SMOKE_TEST=false.

Weight files are 7.55 GB versus 17.53 GB for merged 16-bit: 56.9% smaller, or 2.32× compression.

Quality gate

Complete ChartQA test set, relaxed accuracy with 5% numeric tolerance, isolated vLLM subprocesses:

The predefined maximum degradation was 2 pp. Paired bootstrap 95% CI for the overall change was [-1.40, -0.04] pp.

Public-equivalent weight revision: 43b71926a1d645133560347787539729bcd3de6b.

A100 vLLM benchmark

Run v2-aa4442870cfd; one A100-SXM4-40GB; vLLM 0.25.1+cu129; same 64 ChartQA requests per level; 64 warmup + 64 measured requests; fixed 64-token output; all 8 levels completed with zero failures.

AWQ lowered TPOT p95 and E2E p95 at every tested concurrency. The tradeoff is TTFT p95 at concurrency 4/8/16, which increased by 9.0%/20.9%/18.8%.

Full artifacts: bench/benchmark_results.json, bench/benchmark_table.md, and eval_quant/results.json.

vLLM usage

Use Linux with the verified CUDA 12.9 stack. Install vLLM before importing torch:

pip install -U uv
uv pip install --system --no-cache \
  "vllm==0.25.1+cu129" "torch==2.11.0+cu129" \
  "torchvision==0.26.0+cu129" "torchaudio==2.11.0+cu129" \
  --extra-index-url https://wheels.vllm.ai/0.25.1/cu129 \
  --extra-index-url https://download.pytorch.org/whl/cu129 \
  --index-strategy unsafe-best-match

import base64
from pathlib import Path

from vllm import LLM, SamplingParams

MODEL = "steven0226/qwen3vl-8b-chartqa-awq"
REVISION = "43b71926a1d645133560347787539729bcd3de6b"

image_path = Path("chart.png")
image_uri = (
    "data:image/png;base64," + base64.b64encode(image_path.read_bytes()).decode()
)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": image_uri}},
            {
                "type": "text",
                "text": "Which category has the highest value?\nAnswer the question using a single word or phrase.",
            },
        ],
    }
]

llm = LLM(
    model=MODEL,
    revision=REVISION,
    max_model_len=4096,
    limit_mm_per_prompt={"image": 1, "video": 0},
)
params = SamplingParams(temperature=0, max_tokens=64)
answer = llm.chat([messages], sampling_params=params)[0].outputs[0].text
print(answer)

Intended use

• Chart question answering and controlled chart-reading experiments
• GPU serving with vLLM on compatible NVIDIA hardware
• Research and portfolio demonstrations of QLoRA → AWQ deployment

Limitations

• Primarily evaluated on English ChartQA short answers; not validated for general documents or safety-critical use.
• Fine-tuning gains were concentrated on the augmented subset.
• Benchmark outputs were forced to 64 tokens; each level had only 64 measured requests, so p95 is exploratory.
• vLLM's ~36 GB reserved memory in this run was controlled by gpu_memory_utilization=0.88 and is not evidence of reduced AWQ VRAM reservation.
• Production deployment should add repeated trials, realistic output lengths, and soak testing.

中文摘要

這是 ChartQA 微調後 Qwen3-VL-8B 的 AWQ W4A16 g32 部署版本。語言模型 Linear 權重採 4-bit symmetric group quantization；vision tower 與 lm_head 維持原精度。完整 2,500 題評估為 85.52%，相較 merged 16-bit 下降 0.72 個百分點，通過預設 2 pp 品質門檻。權重檔由 17.53 GB 降至 7.55 GB；A100 vLLM benchmark 在所有測試 concurrency 都提升吞吐並降低 TPOT/E2E p95，但高併發 TTFT p95 較高。

Source repository

程式碼、訓練與評估流程、測試與完整證據都在 GitHub：https://github.com/kuotunyu/qwen3-vl-chartqa。GitHub kuotunyu 與 Hugging Face steven0226 為同一人。

Source code, the training/evaluation pipeline, tests and the full evidence trail live at https://github.com/kuotunyu/qwen3-vl-chartqa. GitHub kuotunyu and Hugging Face steven0226 are the same author.