hf-inference-endpoints

Hugging Face Inference Endpoints Deployment Project

This project demonstrates how to deploy a fine-tuned model to Hugging Face Inference Endpoints, making it accessible via a REST API.

What Was Done

1. Deployment Setup: Created scripts and documentation for deploying models to Hugging Face Inference Endpoints, which provides a managed, scalable way to serve models in production.

2. Endpoint Management: Implemented functionality to:

   • Create new Inference Endpoints
   • Check endpoint status
   • Test deployed endpoints
   • Delete endpoints when no longer needed

3. Testing: Created a test script to make requests to deployed endpoints and verify they're working correctly.

Prerequisites

• Hugging Face account (Pro account required for Inference Endpoints)
• A model pushed to the Hugging Face Hub (from Project 1)
• Hugging Face API token with Inference Endpoints permissions

Files

• deploy_endpoint.py: Script to create and manage Inference Endpoints
• test_endpoint.py: Script to test a deployed endpoint
• README.md: This file

Deployment Methods

Method 1: Using Hugging Face Web Interface (Recommended)

1. Navigate to Inference Endpoints:

   • Go to https://huggingface.co/inference-endpoints
   • Click "New Endpoint"

2. Configure Endpoint:

   • Name: Choose a name for your endpoint (e.g., imdb-sentiment-classifier)
   • Model: Select your model from the Hub (e.g., your-username/imdb-sentiment-distilbert)
   • Cloud Provider: Choose AWS, Azure, or GCP
   • Region: Select a region close to your users
   • Instance Type:
     • CPU-2: For testing and low traffic
     • GPU-T4-Small: For moderate traffic
     • GPU-T4-Medium or larger: For high traffic
   • Task: Select "Text Classification" (or appropriate task)

3. Deploy:

   • Click "Create Endpoint"
   • Wait for deployment (usually 2-5 minutes)
   • Once status shows "Running", your endpoint is ready

4. Get Endpoint URL:

   • Copy the endpoint URL from the dashboard
   • It will look like: https://xxxxx.us-east-1.aws.endpoints.huggingface.cloud

Method 2: Using Python API

from huggingface_hub import HfApi

api = HfApi()

endpoint = api.create_inference_endpoint(
    name="imdb-sentiment-classifier",
    repository="your-username/imdb-sentiment-distilbert",
    framework="pytorch",
    task="text-classification",
    instance_type="cpu-2",
    region="us-east-1",
    vendor="aws",
)

See deploy_endpoint.py for a complete example.

Testing the Endpoint

Using Python

import requests

endpoint_url = "https://your-endpoint-url"
headers = {
    "Authorization": "Bearer YOUR_HF_TOKEN",
    "Content-Type": "application/json"
}

payload = {
    "inputs": "This movie was fantastic!"
}

response = requests.post(f"{endpoint_url}/", headers=headers, json=payload)
print(response.json())

Or use the provided test script:

export HF_ENDPOINT_URL="https://your-endpoint-url"
export HF_TOKEN="your-token"
python test_endpoint.py

Using cURL

curl https://your-endpoint-url/ \
  -X POST \
  -H "Authorization: Bearer YOUR_HF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"inputs": "This movie was fantastic!"}'

Endpoint Management

Check Status

from huggingface_hub import HfApi

api = HfApi()
endpoints = api.list_inference_endpoints()
for endpoint in endpoints:
    print(f"{endpoint.name}: {endpoint.status}")

Update Endpoint

You can update instance type, scale, or other settings through the web interface or API.

Delete Endpoint

Via Web Interface:

• Go to Inference Endpoints dashboard
• Click on your endpoint
• Click "Delete"

Via API:

api.delete_inference_endpoint("imdb-sentiment-classifier")

Cost Considerations

• Inference Endpoints are billed per hour based on instance type
• CPU instances are cheaper but slower
• GPU instances are faster but more expensive
• Consider using CPU for development/testing
• Use GPU for production workloads with high traffic

Best Practices

1. Start Small: Begin with a CPU instance for testing
2. Monitor Usage: Track your endpoint usage and costs
3. Scale Appropriately: Upgrade instance type if you need more throughput
4. Delete Unused Endpoints: Always delete endpoints you're not using to avoid charges
5. Use Environment Variables: Store your endpoint URL and token securely
6. Error Handling: Implement retry logic and error handling in your client code

Troubleshooting

• Endpoint not starting: Check that your model is compatible and properly uploaded
• Authentication errors: Verify your HF token is correct and has proper permissions
• Timeout errors: Consider upgrading to a GPU instance for faster inference
• Rate limiting: Implement exponential backoff in your client code

Example Response

For a text classification endpoint, the response will look like:

[
  {
    "label": "POSITIVE",
    "score": 0.9998
  }
]

Next Steps

• Set up monitoring and logging
• Implement caching for frequently requested predictions
• Create a client library for easier integration
• Set up CI/CD for automatic deployments