medgemma-27b-text-it-FP8-Dynamic

MedGemma-27B-Text-IT-FP8-Dynamic

Overview

MedGemma-27B-Text-IT-FP8-Dynamic is an FP8 Dynamic–quantized derivative of Google’s MedGemma-27B-Text-IT model, optimized for high-throughput inference while preserving strong performance on medical and biomedical instruction-tuned text-only tasks.

This version is intended for vLLM deployment on modern NVIDIA GPUs and follows a conservative FP8 Dynamic quantization strategy designed for maximum stability.

---

Base Model

• Base model: google/medgemma-27b-text-it
• Architecture: Decoder-only Transformer (instruction-tuned)
• Domain: Medical / Biomedical NLP
• Modality: Text-only

---

Quantization Details

• Method: FP8 Dynamic
• Tooling: llmcompressor
• Quantized layers: Linear layers
• Excluded components:
  • lm_head

Rationale

• FP8 Dynamic reduces VRAM usage and improves inference throughput.
• Excluding lm_head preserves output stability.
• The resulting model is fully compatible with vLLM.

Weights are already quantized — do not apply runtime quantization.

---

Intended Use

• Medical and biomedical instruction-following
• Clinical text summarization
• Medical RAG pipelines
• Decision-support and research assistance

---

Deployment (vLLM)

Recommended

vllm serve ig1/medgemma-27b-text-it-FP8-Dynamic \
  --served-model-name medgemma-27b-text-it-fp8 \
  --dtype auto