medgemma-27b-text-it-GPTQ-4bit

MedGemma 27B Text-IT (4-bit GPTQ)

This repository contains a 4-bit GPTQ quantized version of google/medgemma-27b-text-it, optimized for high-throughput inference using vLLM and the Marlin kernel.

MedGemma 27B is a text-only, instruction-tuned medical Large Language Model developed by Google and the Health AI Developer Foundation. It is trained on extensive medical datasets to excel in clinical reasoning, medical Q&A, and healthcare text comprehension.

📊 Quantization Specifications

This model was quantized using GPTQModel with the following parameters to ensure maximum compatibility and speed on modern NVIDIA GPUs:

• Precision: 4-bit (INT4) weights, bfloat16 activations
• Group Size: 128
• Desc Act (Activation Reordering): False (Specifically optimized for vLLM's Marlin kernels to prevent latency penalties and maximize tokens-per-second)
• Calibration Dataset: Medical domain text and general knowledge prompts to preserve clinical reasoning capabilities.

💻 Hardware Requirements

Thanks to 4-bit quantization, the VRAM footprint is drastically reduced compared to the original 16-bit model:

• Model Weights: ~15 GB VRAM
• Recommended GPU: Any NVIDIA GPU with 16GB+ VRAM (e.g., RTX 4080, RTX 4090, L4, L40S, A100) for comfortable context windows up to 8K tokens.
• Compute Capability: Requires a GPU with Compute Capability >= 8.0 (Ampere, Ada Lovelace, Hopper) for optimal Marlin kernel acceleration.

🚀 How to Use

1. High-Throughput Production Serving (Recommended via vLLM)

For the best tokens-per-second (TPS) and concurrent request handling, use vLLM.

Note: Gemma 3 architectures strictly require bfloat16 activations to prevent numerical instability. Do not use float16.

Install vLLM:

pip install vllm

Start the OpenAI-compatible API Server:

python -m vllm.entrypoints.openai.api_server \
    --model HarshBhanushali7705/medgemma-27b-text-it-GPTQ-4bit \
    --served-model-name medgemma-27b-gptq \
    --dtype bfloat16 \
    --quantization gptq \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.90 \
    --port 8000

Query via Python (OpenAI Client):

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="medgemma-27b-gptq",
    messages=[
        {"role": "system", "content": "You are a helpful medical assistant."},
        {"role": "user", "content": "What are the diagnostic criteria for systemic lupus erythematosus?"}
    ],
    temperature=0.0,
    max_tokens=512
)

print(response.choices[0].message.content)

2. Local Inference via Transformers

If you prefer running the model locally in a Python script without a server:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "HarshBhanushali7705/medgemma-27b-text-it-GPTQ-4bit"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16
)

messages = [
    {"role": "system", "content": "You are a helpful medical assistant."},
    {"role": "user", "content": "Explain the pathophysiology of type 2 diabetes mellitus."}
]

inputs = tokenizer.apply_chat_template(
    messages, 
    add_generation_prompt=True, 
    return_tensors="pt"
).to(model.device)

outputs = model.generate(inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

⚠️ Limitations & Medical Disclaimer

This model is for research and development purposes only.

• Not for Clinical Use: The outputs generated by this model are not intended to directly inform clinical diagnosis, patient management decisions, treatment recommendations, or any other direct clinical practice applications.
• Hallucinations: Like all LLMs, MedGemma can hallucinate or provide medically inaccurate information. All outputs require independent verification and clinical correlation by a licensed medical professional.
• Quantization Degradation: While 4-bit GPTQ preserves the vast majority of the base model's intelligence, slight degradation in complex multi-step medical reasoning may occur compared to the original bfloat16 weights.

🙏 Credits & Base Model

This quantized model is built upon the incredible work by Google and the Health AI Developer Foundation.

• Original Base Model: google/medgemma-27b-text-it
• Please refer to the original model card for comprehensive details on the training data, architecture (Gemma 3), and ethical evaluations.

Citation

@misc{medgemma-hf,
  author = {Google},
  title = {MedGemma Hugging Face},
  howpublished = {\url{https://huggingface.co/collections/google/medgemma-release-680aade845f90bec6a3f60c4}},
  year = {2025},
  note = {Accessed: [Insert Date]}
}

👤 Author & Contact

This quantization and vLLM optimization pipeline was created and maintained by:

Harsh Bhanushali

• 🌐 Website: harshbhanushali.in
• 📝 Blog: Compressing MedGemma 27B into 15GB
• 📧 Email: [email protected]
• 🤗 Hugging Face: HarshBhanushali7705

If you find this quantized model useful for your medical AI research or deployment, please consider upvoting the model and citing the original Google MedGemma paper/repository!