flash-attn-windows-blackwell

Flash-Attention Prebuilt Wheels for Windows + Blackwell (SM 12.0)

🎯 Overview

Prebuilt flash-attn wheels for Windows, targeting NVIDIA RTX 50 series (Blackwell / SM 12.0) GPUs, compiled with CUDA 13.0.

🇨🇳 国内镜像 / Also on ModelScope: https://www.modelscope.cn/models/wxd11011/flash-attn-windows-blackwell

This repository fills that gap by compiling from source, making it ideal for ComfyUI / Stable Diffusion / Flux / LLM inference users who want to download and use directly without setting up a build environment.

Two versions are provided — choose based on your installed PyTorch and Python:

---

⚠️ How to Choose

Wheels are version-sensitive. Check your environment first:

python -c "import torch, sys; print('Python:', sys.version); print('PyTorch:', torch.__version__, '| CUDA:', torch.version.cuda)"

• Python 3.13 + PyTorch 2.13.0+cu130 → Version 🅰️ (flash-attn 2.8.4)
• Python 3.12 + PyTorch 2.9.1+cu130 → Version 🅱️ (flash-attn 2.8.3)

---

📦 Download & Install

Method 1: huggingface-cli

# Version 🅰️
huggingface-cli download YOUR_USERNAME/flash-attn-windows-blackwell \
  flash_attn-2.8.4+cu13torch2.13cxx11abiTRUE-cp313-cp313-win_amd64.whl \
  --local-dir ./

# Version 🅱️
huggingface-cli download YOUR_USERNAME/flash-attn-windows-blackwell \
  flash_attn-2.8.3+cu13torch2.9.1cxx11abiTRUE-cp312-cp312-win_amd64.whl \
  --local-dir ./

Method 2: Python API

from huggingface_hub import hf_hub_download

path = hf_hub_download(
    repo_id="YOUR_USERNAME/flash-attn-windows-blackwell",
    filename="flash_attn-2.8.4+cu13torch2.13cxx11abiTRUE-cp313-cp313-win_amd64.whl"
)
print(path)

Method 3: Direct Download

Click the "Files and versions" tab above, then click the download icon next to the .whl file.

Install

pip install flash_attn-2.8.4+cu13torch2.13cxx11abiTRUE-cp313-cp313-win_amd64.whl

ComfyUI portable users, use the embedded Python path:

d:\your_path\ComfyUI_windows_portable_nvidia\python_embeded\python.exe -m pip install the_wheel_file.whl

---

✅ Verify Installation

import torch
from flash_attn import flash_attn_func
import flash_attn

def test_flash_attn():
    print(f"flash_attn version: {flash_attn.__version__}")

    if not torch.cuda.is_available():
        print("❌ CUDA is not available. Flash Attention requires GPU support.")
        return False

    # Small dimensions to avoid OOM
    batch_size, seq_len, n_heads, head_dim = 2, 8, 4, 32
    dtype = torch.float16
    device = "cuda"

    # Random q, k, v with gradients
    q = torch.randn((batch_size, seq_len, n_heads, head_dim), dtype=dtype, device=device, requires_grad=True)
    k = torch.randn((batch_size, seq_len, n_heads, head_dim), dtype=dtype, device=device, requires_grad=True)
    v = torch.randn((batch_size, seq_len, n_heads, head_dim), dtype=dtype, device=device, requires_grad=True)

    # Forward pass
    out = flash_attn_func(q, k, v, causal=False)

    # Check output shape
    expected = (batch_size, seq_len, n_heads, head_dim)
    if out.shape != expected:
        print(f"❌ Output shape mismatch: {out.shape} != {expected}")
        return False

    # Backward pass (test gradients)
    loss = out.sum()
    loss.backward()

    if q.grad is None or k.grad is None or v.grad is None:
        print("❌ Gradient computation failed")
        return False

    print("✅ Flash Attention test passed!")
    return True

if __name__ == "__main__":
    test_flash_attn()

---

🎮 Supported GPU Architectures

Version 🅰️ (flash-attn 2.8.4)

5 architectures × 96 kernels (480 cubins) + 96 sm_120 PTX:

Version 🅱️ (flash-attn 2.8.3)

4 architectures × 72 kernels (288 cubins), no PTX:

❌ NOT Supported

• sm_86: RTX 3060 / 3070 / 3080 / 3080 Ti / A10 / A40
• sm_89: RTX 4060 / 4070 / 4080 / 4090 / L40 / L40S

---

📌 Use Cases

• ✅ ComfyUI custom nodes requiring flash-attn
• ✅ Stable Diffusion / Flux inference acceleration
• ✅ LLM inference (vLLM / Transformers)
• ✅ Any project needing Flash Attention on Windows + Blackwell GPU

---

⚠️ Notes

1. Versions must match exactly, especially PyTorch and Python versions
2. This is a community-compiled, unofficial build — verify file integrity before use
3. If you get DLL load failed, ensure CUDA 13.0 runtime and VC++ 2022 redistributable are installed
4. PyTorch distributed on Windows requires gloo backend (NCCL not supported)

---

🙏 Acknowledgments

• Flash-Attention: https://github.com/Dao-AILab/flash-attention
• PyTorch: https://pytorch.org
• CUDA Toolkit: https://developer.nvidia.com/cuda-toolkit
• Compilation reference: https://blog.csdn.net/u014451778/article/details/156356011

---

📄 License

This is a compiled artifact of flash-attention, licensed under the BSD 3-Clause "New" or "Revised" License.

• Original copyright: Copyright (c) 2022, Tri Dao and flash-attention contributors
• License text: https://github.com/Dao-AILab/flash-attention/blob/main/LICENSE