clair-health

clair-health

Clair is a multimodal medical AI assistant with multimodal text and vision capabilities for Xray/CTScan and disease detection.

It is grounded in:

• Zimbabwe’s public health system, including the Ministry of Health and Child Care, central hospitals, key regulators, and major health indicators.
• Zimbabwean heritage and culture, including Great Zimbabwe, Victoria Falls, national symbols, and local languages.

Important: Clair is for informational and research use only and is not a substitute for professional medical advice, diagnosis, or treatment.

Multimodal vision capability

Clair's text part decoder-only transformer (LLM). The image part is a SigLIP image encoder

This repository includes:

• model-Q4_K_M.gguf — quantized language model for CPU inference.
• mmproj-model-f16.gguf — matching vision projector converted from this fine-tuned model so image understanding works correctly.

Running with Ollama

Create a Modelfile like this:

FROM ./model-Q4_K_M.gguf
CLIP_MODEL ./mmproj-model-f16.gguf
SYSTEM You are Clair, a warm and knowledgeable AI health assistant developed by Michael Nkomo, an AI engineer based in Zimbabwe. You are grounded in Zimbabwe's public health system, Zimbabwean heritage and culture, and Cimas Health Group. You are not a substitute for professional medical advice, diagnosis, or treatment.
PARAMETER temperature 0.7
PARAMETER top_k 40
PARAMETER top_p 0.9

Then run:

ollama create clair-health -f Modelfile
ollama run clair-health

Example prompt:

What's in this image? /path/to/image.jpg

Input and output

• Input: text prompts and images.
• Image handling: images are normalized to 896 × 896 resolution and encoded to 256 tokens each.
• Context length: up to 128K input tokens.
• Output: generated text for answering questions, analyzing image content, or summarizing documents.
• Output length: up to 8192 tokens.

Benchmarks

Multimodal health benchmarks

Text health benchmarks

Chest X-ray report generation

Evaluated on MIMIC-CXR using RadGraph F1.

Safety and ethics

Clair has been evaluated with structured safety testing and internal red-teaming across text and image inputs. These evaluations covered child safety, content safety, representational harms, and medical safety concerns.

The model was tested without safety filters to better understand raw behavior during evaluation. Results should be interpreted carefully, especially because the evaluation set was primarily English-language prompts.

Using Clair-health with Transformers

First, install the Transformers library. Clair-health is supported starting from transformers 4.50.0.

pip install -U transformers

Run the model with the pipeline API

from transformers import pipeline
from PIL import Image
import requests
import torch

pipe = pipeline(
    "image-text-to-text",
    model="clair-health",
    torch_dtype=torch.bfloat16,
    device="cuda",
)

# Image attribution: Stillwaterising, CC0, via Wikimedia Commons
image_url = "https://upload.wikimedia.org/wikipedia/commons/c/c8/Chest_Xray_PA_3-8-2010.png"
image = Image.open(requests.get(image_url, headers={"User-Agent": "example"}, stream=True).raw)

messages = [
    {
        "role": "system",
        "content": [{"type": "text", "text": "You are an expert radiologist."}]
    },
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this X-ray"},
            {"type": "image", "image": image}
        ]
    }
]

output = pipe(text=messages, max_new_tokens=200)
print(output["generated_text"][-1]["content"])

Run the model directly

pip install accelerate

from transformers import AutoProcessor, AutoModelForImageTextToText
from PIL import Image
import requests
import torch

model_id = "clair-health"

model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(model_id)

# Image attribution: Stillwaterising, CC0, via Wikimedia Commons
image_url = "https://upload.wikimedia.org/wikipedia/commons/c/c8/Chest_Xray_PA_3-8-2010.png"
image = Image.open(requests.get(image_url, headers={"User-Agent": "example"}, stream=True).raw)

messages = [
    {
        "role": "system",
        "content": [{"type": "text", "text": "You are an expert radiologist."}]
    },
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this X-ray"},
            {"type": "image", "image": image}
        ]
    }
]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device, dtype=torch.bfloat16)

input_len = inputs["input_ids"].shape[-1]

with torch.inference_mode():
    generation = model.generate(**inputs, max_new_tokens=200, do_sample=False)
    generation = generation[input_len:]

decoded = processor.decode(generation, skip_special_tokens=True)
print(decoded)

License and disclaimer

This model is provided for informational and research purposes only. It is an AI assistant, not a substitute for professional medical advice, diagnosis, or treatment.