ocr-mt-asr-5090

OCR + Qwen3-MT + Cohere ASR on one RTX 5090

Serve kit for three colocated vLLM processes plus a FastAPI gateway with Swagger. This repo is scripts + engines, not a weight dump — vLLM still pulls the official model repos.

Proven together on a Vast.ai 5090 (bf16, no quantization):

Start Qwen first, then OCR, then Cohere. Utils sum to 0.90 so the 32 GB card does not OOM. Do not raise both MT and OCR so they sum past ~0.90 with ASR. Do not switch to 4-bit.

Pull

hf download mohameddalii/ocr-mt-asr-5090 --local-dir ./ocr-mt-asr-5090
cd ocr-mt-asr-5090

Generic 5090 (any Linux box)

python3 -m venv .venv && source .venv/bin/activate
uv pip install -r requirements.txt
# Blackwell: if CUDA tensors fail, reinstall torch with cu128+
# uv pip install torch --index-url https://download.pytorch.org/whl/cu128
uv pip install librosa soundfile   # Cohere audio

bash scripts/prefetch.sh          # optional, caches the three models
bash scripts/serve_three.sh       # MT → OCR → ASR
python3 python/doc_api.py         # gateway on 127.0.0.1:18000

Internal binds:

python python/translate.py --from English --to Arabic "The total is 12 dollars."
python python/ocr.py scan.png
python python/ingest.py invoice.pdf --json

Vast.ai 5090

Need three free normal ports at instance create time (defaults 10100, 10200, 3000) plus the image’s existing 8000 for the gateway.

cd /workspace
hf download mohameddalii/ocr-mt-asr-5090 --local-dir /workspace/ocr-mt-asr-5090
bash /workspace/ocr-mt-asr-5090/vast/install_vast.sh

That copies supervisor wrappers, adds portal entries, starts MT → OCR → ASR, then the gateway. A Cloudflare quick tunnel (*.trycloudflare.com) is started in front of the gateway so the browser padlock is a real public CA. The Vast IP HTTPS cert is self-signed (jupyter.vast.ai) — do not use it from the backend.

Public URLs: vast-capabilities direct_url for raw models (optional), gateway Swagger via the tunnel printed in /var/log/portal/gateway-tunnel.log.

Backend contract (what the app should call)

Needed fields only. Files are binary. target_lang is an ISO 639-1 id (ar, en, fr) — not "Arabic".

curl -sS https://<gateway>/ingest -F [email protected]
# {"text": "...", "lang": "en", "language": "English", "kind": "pdf", ...}

curl -sS https://<gateway>/translate \
  -H 'Content-Type: application/json' \
  -d '{"content":"The total is 12 dollars.","target_lang":"ar"}'
# {"text": "المجموع هو 12 دولارًا.", "lang": "en"}

Environment knobs

Gateway behavior

• /file2text: keeps the original PDF/DOCX/XLSX/PPTX structure. Article PDFs reflow so words wrap at spaces and stay inside the page margins (no mid-word clip at the right edge). Arabic uses Noto Naskh + HarfBuzz.
• /transcribe: ffmpeg → 16 kHz mono WAV, then 30s chunks. Optional Form language (ar / en). Silence-only chunks are skipped; @@فراغ is stripped.
• /ocr: full-page only. CJK is stripped unless the page is actually Chinese.
• /translate: chunks long text so it fits Qwen’s 2048 context.

Layout

scripts/serve_three.sh     start all three (MT first)
scripts/serve_mt.sh
scripts/serve_ocr.sh
scripts/serve_asr.sh
scripts/prefetch.sh
python/doc_api.py          FastAPI gateway + Swagger
python/layout_export.py    file2text overlay / reflow
python/ocr_engine.py       full-page OCR + CJK filter
python/mt_engine.py        glossary + copy-spans + 2048-safe chunks
python/ingest.py           any file → text + language + ASR chunks
python/ocr.py / translate.py
fonts/                     Noto Naskh Arabic + DejaVu Sans
vast/install_vast.sh       supervisor + portal + CF tunnel

Weights: Qwen/Qwen3-4B-Instruct-2507, PaddlePaddle/PaddleOCR-VL, CohereLabs/cohere-transcribe-arabic-07-2026. Two-model-only kit (no ASR/gateway): mohameddalii/ocr-mt-5090.