realesrgan-256x256-tiles-amdnpu

🚀 Real-ESRGAN 256x256 for 4x Single Image Super-Resolution on AMD AI PC NPU

The Real Enhanced Super-Resolution Generative Adversarial Networks (Real-ESRGAN) model is an AI model that takes an input low-resolution image and creates high or "super-resolution" image. This version of the Real-ESRGAN model has been re-trained from scratch with reduced feature channels and fewer stacked blocks for improved efficiency. It was then quantized from FP32 to INT8 and optimized to run on the AMD AI PC NPU. This model takes an input image and scales it up by 4x.

The "256x256" in the title means that this model works on tile sizes 256x256, but almost any size input image can be upscaled by 4x. The inference pipeline tiles the input image into patches based on the ONNX model’s expected input resolution (with overlap), runs inference on each tile, and then stitches the results back together. A model with a larger tile size would lower the stitching overhead and may contain less boundary artifacts. Figure 1 shows an example of a 4x scaled image.

Figure 1: Input 320x480 scaled up by 4x to 1280x1920 with Real-ESRGAN model running on AMD AI PC NPU. Source: EDSR Benchmark dataset (edsr_benchmark\B100\HR\108005.png).

The original model and architecture can be found on GitHub: xinntao/Real-ESRGAN.

Wang et al. (2018) introduced ESRGAN with the "Residual-in-Residual Dense Block (RRDB), without batch normalization as the basic network building unit." Figure 2 shows their original ESRGAN model architecture and Figure 3 shows their updated Real-ESRGAN (Wang et al., 2021) architecture.

Figure 2: ESRGAN architecture with Residual in Residual Dense Block (RRDB) and removed batched normalization (BN) layers. Image from Fig. 4 of Wang et al. (2018).

Figure 3: Real-ESRGAN architecture "adopts the same generator network as that in ESRGAN. For the scale factor of \(\times2\) and \(\times1\), it first employs a pixel-unshuffle operation to reduce spatial size and re-arrange information to the channel dimension." Image from Fig. 4 of Wang et al. (2021).

⚡ Intended Use

How to Use

📐 Hardware Prerequisites

Before getting started, make sure you meet the minimum hardware and OS requirements:

Getting Started

1. Follow the instructions here to download necessary NPU drivers and Ryzen AI software: Ryzen AI SW Installation Instructions. Please allow for around 30 minutes to install all of the necessary components of Ryzen AI SW.

2. Activate the previously installed conda environment from Ryzen AI (RAI) SW, and set the RAI environment variable to your installation path. Substitute the correct RAI version number for v.v.v, such as 1.7.1.

conda activate ryzen-ai-v.v.v
$Env:RYZEN_AI_INSTALLATION_PATH = 'C:/Program Files/RyzenAI/v.v.v/'

1. Clone the Hugging Face model repository:

git clone https://hf.co/amd/realesrgan-256x256-tiles-amdnpu

Alternatively, you can use the Hugging Face Hub API to download all of the files with Python:

from huggingface_hub import snapshot_download
snapshot_download("amd/realesrgan-256x256-tiles-amdnpu")

1. Install the necessary packages into the existing conda environment:

pip install -r requirements.txt

This covers both inference and the accuracy evaluation further down, in the one environment. ONNX Runtime is not among the dependencies on purpose: the Ryzen AI environment already provides onnxruntime-vitisai, and installing plain onnxruntime over it removes the NPU execution provider without reporting an error. Nothing listed pulls it in, so the NPU stays available for evaluation runs as well as inference.

1. Data Preparation (optional: for evaluation). Download the EDSR benchmark dataset extract it into the datasets/ directory. Note that you will need to run this script twice, as it seems to fail on first attempt.

python download_edsr_benchmark.py

Download and extract the DIV2K validation set:

python download_div2k.py

The datasets/ directory should look like this:

datasets
  └──DIV2K_valid_HR
  └──DIV2K_valid_LR_bicubic/X4
  └──edsr_benchmark
      └── B100
            └── HR
              ├── 3096.png
              ├── ...
            └── LR_bicubic/X4
              ├── 3096x4.png
              ├── ...
      └── Set5
            └── HR
              ├── baby.png
              ├── ...
            └── LR_bicubic/X4
              ├── babyx4.png
              ├── ...

1. Run inference on a single image or a folder of images. For example, for one image, run

python onnx_inference.py --onnx onnx-models/realesrgan_nchw_256x256_u8s8.onnx --input .\datasets\edsr_benchmark\B100\HR\108005.png --out-dir outputs  --device npu

Arguments:

--onnx: The ONNX model file path.

--input: Accepts either a single image file path or a directory path. If it's a file, the script will process that image only. If it's a directory, the script will recursively scan for .png, .jpg, and .jpeg files and process all of them.

--out-dir: Output directory where the restored images will be saved.

--device: Accepts "npu" or "cpu". The NPU will attempt to use the VitisAIExecutionProvider; the CPU will attempt to use the CPUExecutionProvider. Note that to use the NPU, the updated NPU drivers and Ryzen AI SW must first be installed.

The model has already been compiled and cached under modelcachekey_realesrgan_nchw_256x256_u8s8, but if this folder is not present, the model will be recompiled and then inference can be run.

1. Evaluate the accuracy of the model on benchmark datasets (optional).

Eval on Set14. Enabling the -clean option will remove generated SR images.

python onnx_eval.py --onnx onnx-models/realesrgan_nchw_256x256_u8s8.onnx --hq-dir datasets/edsr_benchmark/Set14/HR --lq-dir datasets/edsr_benchmark/Set14/LR_bicubic/X4 --out-dir outputs/u8s8-Set14 --device npu -clean

The output will be a set of accuracy metrics: PSNR, MS_SSIM, SSIM, and FID, in JSON format as below:

{
  "onnx": "onnx-models/realesrgan_nchw_256x256_u8s8.onnx",
  "psnr": 23.29035758972168,
  "ms_ssim": 0.8946605591772246,
  "ssim": 0.6391846092521831,
  "fid": 137.04519039887862
}

The following are example scripts to run evaluation on the other datasets:

Eval on B100:

python onnx_eval.py --onnx onnx-models/realesrgan_nchw_256x256_u8s8.onnx --hq-dir datasets/edsr_benchmark/B100/HR --lq-dir datasets/edsr_benchmark/B100/LR_bicubic/X4 --out-dir outputs/u8s8-B100 --device npu -clean

Eval on Urban100:

python onnx_eval.py --onnx onnx-models/realesrgan_nchw_256x256_u8s8.onnx --hq-dir datasets/edsr_benchmark/Urban100/HR --lq-dir datasets/edsr_benchmark/Urban100/LR_bicubic/X4 --out-dir outputs/u8s8-Urban100 --device npu -clean

Eval on DIV2K:

python onnx_eval.py --onnx onnx-models/realesrgan_nchw_256x256_u8s8.onnx --hq-dir datasets/DIV2K_valid_HR --lq-dir datasets/DIV2K_valid_LR_bicubic/X4 --out-dir outputs/u8s8-DIV2K --device npu -clean

🔧 Evaluation Data

Datasets:

The AMD ONNX model results were evaluated with the DIV2K and EDSR (B100, Urban100, Set14, Set5) datasets on peak signal-to-noise ratio (PSNR), multi-scale structural similarity (MS-SSIM), and Fréchet Inception Distance (FID) (see Table 1).

The original Real-ESRGAN model from Wang et al. (2021) was evaluated on RealSR-Canon, RealSR-Nikon, DRealSR, DPED-iphone, OST300, ImageNet val, and ADE20K val (see Table 2).

Figure 4 shows their perceptual quality results as compared to other state-of-the-art models.

Figure 4: "Qualitative comparisons on several representative real-world samples with upsampling scale factor of 4. Our Real-ESRGAN outperforms previous approaches in both removing artifacts and restoring texture details. Real-ESRGAN+ (trained with sharpened ground-truths) can further boost visual sharpness. Other methods may either fail to remove overshoot (the 1st sample) and complicated artifacts (the 2nd sample), or fail to restore realistic and natural textures for various scenes (the 3rd, 4th, 5th samples)". Image and caption from Fig. 7 of Wang et al. (2021).

The original ESRGAN model from Wang et al. (2018) is evaluated "on widely used benchmark datasets – Set5, Set14, BSD100, Urban100, and the PIRM self-validation dataset that is provided in the PIRM-SR Challenge."

Motivation: We evaluate the model's performance to industry standards datasets in quantitative measures (see Quantitative Analyses).

📚 Training Data

Both the Real-ESRGAN and the the original ESRGAN model were trained on 3 image datasets:

1. DIV2K: a set of 800 2K-resolution images for image restoration tasks.
2. Flickr2K: 2,650 2K-resolution images collected on the Flickr website.
3. OutdoorSceneTraining (OST): 10,324 1K- to 2K-resolution images of outdoor scenes.

However, the Real-ESRGAN data were synthetically generated through a preprocessing workflow to degrade the images involving blur, downsampling, noise, and compression.

Real-ESRGAN is finetuned from ESRGAN for faster convergence, for 400K iterations and a learning rate of \(1\times10^{-4}\). "RealESRGAN is trained with a combination of L1 loss, perceptual loss and GAN loss, with weights {1,1,0.1}, respectively" (Wang et al., 2021). For more detailed information on training, see their paper.

📝 Quantitative Analyses

Table 1 shows the accuracy metrics for the AMD ONNX models of Real-ESRGAN.

Table 1: Model accuracy metrics for AMD AI PC FP32 and INT8 quantized models.

The following are the performance results of 6 super-resolution models on the Strix machine NPU (Table 2).

Table 2: Performance metrics in frames per second (FPS) for AMD Super-Resolution models (higher is better).

In their paper, Wang et al. (2021) compare Real-ESRGAN with several state-of-the-art methods with NIQE scores (Table 3).

Table 3: "NIQE scores on several diverse testing datasets with real-world images. The lower, the better." From Table 1 in Wang et al. (2021).

⚓ Ethical Considerations

AMD is committed to conducting our business in a fair, ethical and honest manner and in compliance with all applicable laws, rules and regulations. You can find out more at the AMD Ethics and Compliance page.

⚠️ Caveats and Recommendations

Wang et al. (2021) note that there are limitations with the Real-ESRGAN model, including aliasing, introduction of unpleasant artifacts, and the inability to remove complicated degradations.

📌 Citation Details

@InProceedings{wang2021realesrgan,
    author    = {Xintao Wang and Liangbin Xie and Chao Dong and Ying Shan},
    title     = {Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data},
    booktitle = {International Conference on Computer Vision Workshops (ICCVW)},
    date      = {2021}
}