llama-guard-3-8b-custom

Llama-Guard-3-8B Custom Handler

Overview

This repository contains a custom handler for deploying the meta-llama/Llama-Guard-3-8B model as a HuggingFace Inference Endpoint.

Model Information

• Base Model: meta-llama/Llama-Guard-3-8B
• Task: Text Classification (Safety Filtering)
• Purpose: Content moderation and safety filtering for the Dx²⁴ AI disease diagnosis system
• Use Case: Guardrail #8 - Input/Output Safety Filtering

Handler Features

• Automatic Device Detection: Uses GPU if available, falls back to CPU
• Optimized Performance: Uses bfloat16 precision on GPU for faster inference
• Flexible Input: Accepts single text strings or batches of text
• Error Handling: Graceful error handling with informative messages
• Normalized Output: Consistent JSON format for classification results

Input Format

{
  "inputs": "Text to classify for safety"
}

Or for batch processing:

{
  "inputs": ["Text 1", "Text 2", "Text 3"]
}

Output Format

[
  {
    "label": "safe",
    "score": 0.95
  }
]

Deployment Instructions

1. Go to HuggingFace Inference Endpoints
2. Click "+ New Endpoint"
3. Configure:
   • Model Repository: sadisylarq/llama-guard-3-8b-custom
   • Endpoint Name: dx24-llama-guard
   • Region: us-east-1
   • Instance Type: nvidia-l4 or nvidia-a10g
   • Min Replica: 0
   • Max Replica: 1
   • Scale to Zero Timeout: 1800 seconds
4. Click "Create Endpoint"

Testing

Once deployed, test with:

import requests

API_URL = "https://your-endpoint-url.aws.endpoints.huggingface.cloud"
headers = {"Authorization": "Bearer YOUR_HF_TOKEN"}

data = {"inputs": "Is this content safe?"}
response = requests.post(API_URL, headers=headers, json=data)
print(response.json())

Requirements

• transformers>=4.30.0
• torch>=2.0.0
• GPU recommended for optimal performance

License

Follows the license of the base model: meta-llama/Llama-Guard-3-8B