resnet50-inference-optimization

ResNet-50 (ONNX) — inference-optimization benchmark

This is a standard torchvision ResNet-50 (IMAGENET1K_V2 weights) exported to ONNX, published as the portable artifact from an inference-optimization study. The point of the repo is not the weights — they're the stock torchvision model — but the benchmark: what FP16, ONNX Runtime, and TensorRT (FP16 / INT8) do to latency and accuracy on the same hardware, measured the same way.

The .onnx file lets you reproduce the ONNX Runtime and TensorRT results, or build your own engine, without re-exporting.

Benchmark summary

NVIDIA GPU, CUDA 12.1, TensorRT 10.13. 200 timed iterations, 20 warm-up discarded, torch.cuda.synchronize() around each timed region. Accuracy on a fixed 3,200-image ImageNet-val slice, identical across variants.

Batch-1 latency (single-request):

Batch-32 throughput:

Headline: TensorRT INT8 is 11.6x faster than FP32 at batch 1 for a 0.28-point top-1 drop.

The accuracy figures are higher than the canonical ~80.3% because they're on an easier 3,200-image slice; the relative gap between variants is the point, and it's valid because every variant saw the same images.

Intended use

Reference artifact for inference-optimization work: a fixed ResNet-50 ONNX graph you can run in ONNX Runtime or compile with TensorRT to reproduce or extend the numbers above. Not a new or improved model — the weights are stock torchvision.

How to use

ONNX Runtime (CUDA):

import onnxruntime as ort
import numpy as np

sess = ort.InferenceSession("resnet50.onnx", providers=["CUDAExecutionProvider"])
x = np.random.randn(1, 3, 224, 224).astype(np.float32)  # NCHW, ImageNet-normalized
out = sess.run(None, {sess.get_inputs()[0].name: x})[0]
pred = out.argmax(1)

Preprocessing is standard ImageNet eval: resize 256, center-crop 224, normalize with mean [0.485, 0.456, 0.406] and std [0.229, 0.224, 0.225], in NCHW order.

The input has a dynamic batch axis, so any batch size works.

Model details

• Architecture: ResNet-50
• Weights: torchvision ResNet50_Weights.IMAGENET1K_V2
• Input: (batch, 3, 224, 224) float32, ImageNet-normalized, NCHW
• Output: (batch, 1000) logits over ImageNet-1k classes
• ONNX opset: 17

Limitations

Batch-1 p95 latency is noisy at sub-millisecond scale and shouldn't be read as a tail-latency guarantee. Absolute accuracy is on a subset, not the full 50k val set. INT8 numbers come from TensorRT 10.x implicit calibration (deprecated in 11.x, which uses explicit QDQ via NVIDIA ModelOpt).

Full code, benchmark scripts, and methodology: github.com/paulsaurav/resnet50-inference-optimization.