LLM3_fb835ecc

flux-schnell-edge-inference

This holds the baseline for the FLUX Schnel NVIDIA GeForce RTX 4090 contest, which can be forked freely and optimized

Some recommendations are as follows:

• Installing dependencies should be done in pyproject.toml, including git dependencies
• HuggingFace models should be specified in the models array in the pyproject.toml file, and will be downloaded before benchmarking
  • The pipeline does not have internet access so all dependencies and models must be included in the pyproject.toml
  • Compiled models should be hosted on HuggingFace and included in the models array in the pyproject.toml (rather than compiling during loading). Loading time matters far more than file sizes
• Avoid changing src/main.py, as that includes mostly protocol logic. Most changes should be in models and src/pipeline.py
• Ensure the entire repository (excluding dependencies and HuggingFace models) is under 16MB

For testing, you need a docker container with pytorch and ubuntu 22.04. You can download your listed dependencies with uv, installed with:

pipx ensurepath
pipx install uv

You can then relock with uv lock, and then run with uv run start_inference