ocr-mt-5090

OCR + Qwen3-4B on one RTX 5090

Serve PaddleOCR-VL and Qwen/Qwen3-4B-Instruct-2507 on the same 32GB 5090 without quantization. This repo is the serve kit (scripts + engines), not a new weight dump — vLLM still pulls the official model repos.

Proven together on a Vast.ai 5090 (bf16):

Start Qwen first, then OCR. Both vLLM processes budget --gpu-memory-utilization against the full 32GB, not remaining memory.

Repo: https://huggingface.co/mohameddalii/ocr-mt-5090

Pull

hf download mohameddalii/ocr-mt-5090 --local-dir ./ocr-mt-5090
cd ocr-mt-5090

Generic 5090 (any Linux box)

python3 -m venv .venv && source .venv/bin/activate
uv pip install -r requirements.txt
# Blackwell: if CUDA tensors fail, reinstall torch with cu128+
# uv pip install torch --index-url https://download.pytorch.org/whl/cu128

bash scripts/prefetch.sh          # optional, caches both models
bash scripts/serve_both.sh        # MT then OCR; leaves both in the foreground wait

Endpoints:

• OCR: http://127.0.0.1:18000/v1
• Translation: http://127.0.0.1:18100/v1

python python/ocr.py scan.png
python python/translate.py --from English --to Arabic "The total is 12 dollars."

Vast.ai 5090

On a Vast base/pytorch image with supervisor + Caddy:

cd /workspace
hf download mohameddalii/ocr-mt-5090 --local-dir /workspace/ocr-mt-5090
bash /workspace/ocr-mt-5090/vast/install_vast.sh

That copies supervisor wrappers, adds PaddleOCR-VL (10100→18000) and Qwen3-MT (10200→18100) to /etc/portal.yaml, starts MT, waits, then starts OCR.

Need two free normal ports at instance create time (defaults 10100 and 10200). Public URLs come from vast-capabilities direct_url plus $OPEN_BUTTON_TOKEN.

Environment knobs

Do not raise both utils so they sum past ~0.90 — the card will OOM. Do not switch to 4-bit; that is what this split avoids.

Layout

scripts/serve_both.sh      start both (MT first)
scripts/serve_mt.sh
scripts/serve_ocr.sh
scripts/prefetch.sh
python/ocr_engine.py       parallel crops + OCR: prompt
python/mt_engine.py        glossary + copy-spans
python/ocr.py / translate.py
vast/install_vast.sh       supervisor + portal