medgemma-4b-it-mxfp8

MedGemma 4B-IT (MXFP8 Microscaling)

This is an MXFP8 (Microscaling 8-bit Float) quantized version of google/medgemma-4b-it created using llm-compressor and formatted in compressed-tensors.

MXFP8 conforms to the OCP Microscaling Formats (MX) Specification, utilizing microscopic block-wise scaling (group_size=32) with E8M0 scale exponents. This architecture delivers superior numerical fidelity compared to standard per-tensor FP8 while achieving native tensor core acceleration on NVIDIA Blackwell (SM 10.0+) architecture.

Quantization Specifications

• Base Model: google/medgemma-4b-it
• Quantization Framework: llm-compressor
• Quantization Scheme: MXFP8
  • Weights: Float8 (E4M3), group-wise scaling (group_size=32), E8M0 scale factors
  • Input Activations: Dynamic group-wise microscaling (group_size=32)
• Preserved Precision (BF16): lm_head, embed_tokens, multi_modal_projector, and vision tower components are kept unquantized to guarantee full clinical and diagnostic fidelity.
• Hardware Platform: Quantized and validated on NVIDIA RTX PRO 6000 Blackwell Server Edition (98 GB VRAM).

High-Performance Deployment with vLLM

vLLM natively parses compressed-tensors MXFP8 checkpoints:

from vllm import LLM, SamplingParams

model_name = "Demondiablo/medgemma-4b-it-mxfp8"

# Initialize vLLM engine
llm = LLM(
    model=model_name,
    trust_remote_code=True,
    max_model_len=4096,
)

prompt = "Analyze the clinical implications of an acute ST-elevation myocardial infarction (STEMI)."
messages = [{"role": "user", "content": prompt}]

sampling_params = SamplingParams(
    temperature=0.2,
    max_tokens=512,
    top_p=0.95,
)

outputs = llm.chat(messages=messages, sampling_params=sampling_params)
print(outputs[0].outputs[0].text)

Checkpoint Files

• model.safetensors: MXFP8 compressed weights with per-group E8M0 scale factors
• config.json: Model architecture with quantization_config metadata
• recipe.yaml: Reproducible LLM Compressor recipe
• Tokenizer, processor, and chat template files for complete offline compatibility