paligemma2-10b-mix-224-int8

rivitti/paligemma2-10b-mix-224-int8

INT8-ready release of PaliGemma2 10B for NVIDIA GPU inference with Transformers + BitsAndBytes.

This repository is intended to be loaded with 8-bit quantization at inference time.

Model Details

• Base model: google/paligemma2-10b-mix-224
• Task type: Vision-Language (VQA / image-text generation)
• Inference target: NVIDIA GPU (tested workflow for A40)
• Quantization runtime: BitsAndBytes (load_in_8bit=True)

Requirements

• Python 3.10+
• torch with CUDA support
• transformers 4.58.0 or newer
• bitsandbytes 0.44.0 or newer
• accelerate
• pillow

Quick Start (INT8 inference)

from transformers import AutoProcessor, PaliGemmaForConditionalGeneration, BitsAndBytesConfig from PIL import Image import torch

repo_id = "rivitti/paligemma2-10b-mix-224-int8"

processor = AutoProcessor.from_pretrained(repo_id) model = PaliGemmaForConditionalGeneration.from_pretrained( repo_id, quantization_config=BitsAndBytesConfig(load_in_8bit=True), device_map="cuda:0", torch_dtype=torch.float16, )

image = Image.open("your_image.jpg").convert("RGB") prompt = "<image> Answer briefly: What is in this image?"

inputs = processor(text=prompt, images=image, return_tensors="pt") inputs = {k: v.to("cuda:0") for k, v in inputs.items()}

with torch.no_grad(): output = model.generate(**inputs, max_new_tokens=32, do_sample=False)

text = processor.batch_decode(output[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0] print(text.strip())

Notes

• Add the <image> token at the beginning of the prompt when using both text and images.
• If you run out of GPU memory, keep CUDA_VISIBLE_DEVICES=0 and close other GPU workloads.
• If loading fails, verify transformers/bitsandbytes versions and CUDA wheel compatibility.

Limitations

• Accuracy can vary depending on prompt format and decoding parameters.
• Short-answer prompting is recommended for VQA benchmarking.

Intended Use

• Research and benchmarking for VLM inference.
• VQA experiments with reproducible 8-bit loading settings.