medgemma-1.5-4b-it-4bit-nf4

MedGemma-1.5-4B-IT — 4-bit NF4 Quantised

This is google/medgemma-1.5-4b-it quantised to 4-bit NF4 using bitsandbytes, generated on Kaggle 2×T4 in AIR-LLM style.

Hardware requirements

Quick start

from transformers import AutoProcessor, AutoModelForImageTextToText, BitsAndBytesConfig
import torch

bnb = BitsAndBytesConfig(
    load_in_4bit=True, bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.float16,
)
processor = AutoProcessor.from_pretrained("aab20abdullah/medgemma-1.5-4b-it-4bit-nf4")
model = AutoModelForImageTextToText.from_pretrained(
    "aab20abdullah/medgemma-1.5-4b-it-4bit-nf4", quantization_config=bnb, device_map="auto"
)
model.eval()

messages = [{"role": "user", "content": [{"type": "text", "text": "What is aspirin?"}]}]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(out[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True))

For educational / research use only. Not medical advice.