gemma-2-9b-custom

Gemma-2-9b Custom Handler

Overview

This repository contains a custom handler for deploying the google/gemma-2-9b model as a HuggingFace Inference Endpoint.

Model Information

• Base Model: google/gemma-2-9b
• Task: Text Generation
• Purpose: Medical diagnosis generation for the Dx²⁴ AI disease diagnosis system
• Use Case: Primary diagnosis model for generating medical assessments

Handler Features

• Automatic Device Detection: Uses GPU if available, falls back to CPU
• Optimized Performance: Uses bfloat16 precision on GPU for faster inference
• Flexible Parameters: Customizable generation parameters (temperature, top_p, max_tokens)
• Medical-Optimized Defaults: Pre-configured parameters for medical text generation
• Error Handling: Graceful error handling with informative messages
• Memory Efficient: Uses low CPU memory usage mode

Input Format

Basic input:

{
  "inputs": "Patient presents with fever and cough. What are potential diagnoses?"
}

With custom parameters:

{
  "inputs": "Patient presents with fever and cough. What are potential diagnoses?",
  "parameters": {
    "max_new_tokens": 512,
    "temperature": 0.7,
    "top_p": 0.9,
    "do_sample": true,
    "repetition_penalty": 1.1
  }
}

Output Format

[
  {
    "generated_text": "Based on the symptoms of fever and cough, potential diagnoses include..."
  }
]

Default Parameters

• max_new_tokens: 512 (maximum tokens to generate)
• temperature: 0.7 (controls randomness, lower = more deterministic)
• top_p: 0.9 (nucleus sampling threshold)
• do_sample: true (enables sampling for varied outputs)
• repetition_penalty: 1.1 (prevents repetitive text)

Deployment Instructions

1. Go to HuggingFace Inference Endpoints
2. Click "+ New Endpoint"
3. Configure:
   • Model Repository: sadisylarq/gemma-2-9b-custom
   • Endpoint Name: dx24-gemma-2-9b
   • Region: us-east-1
   • Instance Type: nvidia-a10g (recommended) or nvidia-l4
   • Min Replica: 0
   • Max Replica: 1
   • Scale to Zero Timeout: 1800 seconds
4. Click "Create Endpoint"

Note: Gemma-2-9b requires more powerful GPUs. The nvidia-a10g instance is recommended for optimal performance.

Testing

Once deployed, test with:

import requests

API_URL = "https://your-endpoint-url.aws.endpoints.huggingface.cloud"
headers = {"Authorization": "Bearer YOUR_HF_TOKEN"}

data = {
    "inputs": "Patient presents with fever and cough. Provide diagnosis.",
    "parameters": {
        "max_new_tokens": 256,
        "temperature": 0.7
    }
}

response = requests.post(API_URL, headers=headers, json=data)
print(response.json())

Requirements

• transformers>=4.30.0
• torch>=2.0.0
• GPU with at least 16GB VRAM recommended (A10G or better)

Performance Notes

• First request may take 30-60 seconds as the model loads
• Subsequent requests are much faster (1-5 seconds depending on output length)
• Scale-to-zero saves costs during idle periods

License

Follows the license of the base model: google/gemma-2-9b