medgemma-4b-it-fp8-w8a8

MedGemma 4B-IT (FP8 W8A8 Dynamic)

This is an FP8 W8A8 Quantized version of google/medgemma-4b-it optimized for high-throughput, low-latency deployment with vLLM and compatible serving engines.

Quantization Details

• Base Model: google/medgemma-4b-it
• Quantization Method: llm-compressor
• Quantization Scheme: FP8_DYNAMIC (Weights in FP8 E4M3, dynamic FP8 activations)
• Format: compressed-tensors (native vLLM format)
• Excluded Modules: lm_head, embed_tokens, multi_modal_projector, and vision tower linear layers preserved for accuracy.
• Quantized On: NVIDIA RTX PRO 6000 Blackwell Server Edition

Deployment & Usage

1. High-Performance Inference with vLLM (Recommended)

from vllm import LLM, SamplingParams

model_name = "Demondiablo/medgemma-4b-it-fp8-w8a8"

llm = LLM(
    model=model_name,
    trust_remote_code=True,
    max_model_len=4096,
)

sampling_params = SamplingParams(
    temperature=0.2,
    max_tokens=2048,
)

prompt = "You are a clinical AI. Summarize the following prescription: Tablet Thyronorm 50 mcg OD."
outputs = llm.generate([prompt], sampling_params)
print(outputs[0].outputs[0].text)

2. Inference with Transformers

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText

model_name = "Demondiablo/medgemma-4b-it-fp8-w8a8"

processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

prompt = "Extract medications: Capsule Desula 21 HS, Thyronorm 50 mg OD."
inputs = processor(text=prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=512)
print(processor.decode(outputs[0], skip_special_tokens=True))

Hardware Compatibility

Native FP8 W8A8 acceleration is supported on:

• NVIDIA Ada Lovelace (RTX 4090, RTX 6000 Ada, etc.)
• NVIDIA Hopper (H100, H200)
• NVIDIA Blackwell (RTX PRO 6000 Blackwell, B100, B200)