flux-dev-handler-h200

FLUX.1-dev Handler for H200 GPU

High-quality image generation using FLUX.1-dev optimized for NVIDIA H200 GPU with FP8 quantization.

Features

• FP8 Quantization: Native FP8 support on Hopper architecture (~1.5-2.4x speedup)
• BF16 Precision: Optimal for H200 Tensor Cores
• torch.compile: CUDA graph optimization (~1.3x speedup)
• VAE Slicing: Memory-efficient batch processing
• Large Batch Size: H200's 141GB VRAM enables 16+ images in parallel

Performance

GPU Requirements

API Usage

Single Image Generation

{
  "inputs": "A professional portrait of a business person in a modern office",
  "parameters": {
    "num_inference_steps": 28,
    "guidance_scale": 3.5,
    "width": 1344,
    "height": 768
  }
}

Batch Generation

{
  "inputs": [
    "A sunset over mountains",
    "A city skyline at night",
    "A peaceful forest scene"
  ],
  "parameters": {
    "num_inference_steps": 28,
    "guidance_scale": 3.5
  }
}

Response Format

Single:

{
  "image": "data:image/png;base64,..."
}

Batch:

[
  {"image": "data:image/png;base64,..."},
  {"image": "data:image/png;base64,..."}
]

Parameters

Environment Variables

FP8 Control

• auto (default): Enable FP8 for GPUs with Compute Capability >= 8.9
• true: Force enable FP8 (will fail on unsupported GPUs)
• false: Disable FP8 (use BF16 only)

Deployment

1. Create a new model repository on Hugging Face Hub
2. Upload handler.py and requirements.txt
3. Create a Dedicated Endpoint with H200/H100/L40S GPU
4. Set Task type to "Custom"
5. Add HF_TOKEN secret for gated model access

Comparison with Other Handlers

Technical Details

FP8 Quantization

FP8 (8-bit floating point) quantization is applied to the transformer weights using torchao:

from torchao.quantization import quantize_, float8_weight_only
quantize_(pipe.transformer, float8_weight_only())

Benefits:

• ~50% memory reduction for transformer weights
• ~1.5-2.4x inference speedup on Hopper GPUs
• Minimal quality loss (typically imperceptible)

Requirements:

• Compute Capability >= 8.9 (H200, H100, L40S, RTX 4090)
• torchao >= 0.5.0

VAE Slicing

VAE slicing is enabled for memory-efficient batch processing:

pipe.enable_vae_slicing()

This processes images one at a time during VAE decoding, reducing peak memory usage.

References

• FLUX.1-dev
• torchao
• Diffusers FluxPipeline