vllm-jetson-orin

vLLM 0.19.0 for Jetson Orin (CUDA 12.6, SM 8.7)

Pre-built vLLM 0.19.0 wheel for NVIDIA Jetson Orin family — ready within 72 hours of Gemma 4 release.

Highlights

• vLLM 0.19.0 — latest release with native Gemma 4 architecture support
• SM 8.7 tensor cores — compiled specifically for Jetson Orin
• CUDA 12.6 / JetPack 6.2 — matches current Jetson SDK
• 529 MB wheel — single file install, no compilation needed

Supported Devices

Install (wheel)

# Create venv
python3 -m venv ~/vllm-env && source ~/vllm-env/bin/activate

# Install PyTorch for Jetson (bundled in this repo)
pip install https://huggingface.co/YuyiRobot/vllm-jetson-orin/resolve/main/torch-2.10.0-cp310-cp310-linux_aarch64.whl

# Install vLLM
pip install https://huggingface.co/YuyiRobot/vllm-jetson-orin/resolve/main/vllm-0.19.0+cu126-cp310-cp310-linux_aarch64.whl

# Install transformers for Gemma 4
pip install --no-deps "transformers>=5.5.0"

Run Gemma 4

vllm serve /path/to/gemma-4-E4B-it-W4A16 \
    --host 0.0.0.0 --port 8000 \
    --served-model-name gemma-4-e4b \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.65 \
    --enable-prefix-caching

Benchmark (Jetson AGX Orin 64GB)

Model: Gemma-4-E4B-IT-W4A16 (GPTQ W4A16)

Docker results: Decode 28.3 tok/s, Prefill 563.4 tok/s (similar performance)

Useful Links

• Docker image (recommended for production): ghcr.io/yuyirobotlab/vllm-orin:0.19.0
• Build from source

Acknowledgements

• vLLM — the inference engine
• thehighnotes/vllm-jetson-orin — pioneered vLLM wheel distribution for Jetson Orin
• NVIDIA Jetson AI Lab — PyTorch wheels and ecosystem