Biomni-R0-32B-FP8

Biomni-R0-32B-FP8

This is an FP8 quantized version of Biomni-R0-32B-Preview, optimized for NVIDIA H100 and L40S hardware acceleration.

Quantization Details

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "hassanshka/Biomni-R0-32B-FP8",
    device_map="auto",
    torch_dtype="auto",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("hassanshka/Biomni-R0-32B-FP8")

# Inference
messages = [{"role": "user", "content": "Your medical question here"}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0]))

Quantization Script

from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor import oneshot

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8",
    ignore=["lm_head"]
)

oneshot(
    model=model,
    dataset=calibration_data,
    recipe=recipe,
    max_seq_length=4096,
    num_calibration_samples=len(calibration_data),
)

Performance

• Memory Reduction: ~50% compared to BF16
• Inference Speed: 2-3x faster on H100/L40S with FP8 Tensor Cores
• Accuracy: Near-lossless compared to BF16

Hardware Requirements

⚠️ Requires NVIDIA H100, L40S, or Ada Lovelace GPUs for optimal FP8 performance.

License

Apache 2.0 (same as base model)

Citation

If you use this model, please cite the original Biomni model.