RIFE_fp32_timestep

RIFE_fp32_timestep

FuryTMP/RIFE_fp32 with its baked timestep exposed as a runtime input, so the model can synthesise a frame at any phase between two inputs instead of only the midpoint.

Nothing about the weights changed. The output at t = 0.5 is bit-identical to the original model.

Why

RIFE v4.x IFNet is timestep-conditioned: it takes a constant-valued t plane alongside the two frames and feeds it to the head of every IFBlock. The upstream ONNX export was traced with timestep = 0.5, which folded that value into a Constant node and left the graph with a 6-channel input. The capability was still in the graph, just unreachable — passing a 7th channel changed the output by exactly 0.0.

Internally the plane is built as:

t_plane = ((Slice_4 * 0) + 1) * Constant_50   where Constant_50 = 0.5

This model deletes /Constant_50 and rewires its single consumer (/Mul_3) to a new scalar graph input named timestep. See patch_rife_timestep.py, which reproduces this file byte-for-byte from the upstream RIFE_fp32.onnx.

Inputs / outputs

H and W are arbitrary — the graph pads to a multiple of 32 internally and crops back.

The endpoints are not exact. t = 0 does not reproduce frame0 and t = 1 does not reproduce frame1. This is a property of the upstream weights, not of the patch. It is harmless for interpolation, where the endpoints are the source frames and are passed through verbatim. Use 0 < t < 1.

Usage

import numpy as np, onnxruntime as ort

sess = ort.InferenceSession("RIFE_fp32_timestep.onnx")

x = np.zeros((1, 6, H, W), np.float32)
x[0, 0:3] = frame0          # planar RGB, [0, 1]
x[0, 3:6] = frame1

# timestep must be a 0-d array; a bare np.float32 scalar raises
# "Unable to handle object of type <class 'numpy.float32'>"
out = sess.run(None, {"input": x, "timestep": np.array(0.25, dtype=np.float32)})[0][0]

For a fully-GPU graph under onnxruntime-web's WebGPU EP, pin the free dimensions at session creation. Without this, the meshgrid construction falls back to the CPU execution provider:

await ort.InferenceSession.create(bytes, {
  executionProviders: ["webgpu"],
  graphOptimizationLevel: "all",
  freeDimensionOverrides: { dynamic_dim_0: 1, dynamic_dim_1: 6, dynamic_dim_2: H, dynamic_dim_3: W },
})

With those overrides the graph folds to 336 nodes and zero shape-computation ops — the same node count as the unpatched model. Exposing the timestep costs nothing.

Quality

Midpoint reconstruction against ground truth, 720p, 48 frames from three Xiph derf sequences (park_joy, old_town_cross, in_to_tree). Each dropped frame is rebuilt from its two neighbours and compared to the original.

At non-midpoint phases (gaps of 3, 4 and 8) this model wins at every phase on every clip by 0.4–0.9 dB. Reaching a given phase directly versus by recursive halving measures within ±0.2 dB with no consistent winner, so prefer direct — the intermediates are then independent of one another and can be evaluated in any order.

License and attribution

MIT, inherited unchanged.

• FuryTMP/RIFE_fp32 — the ONNX export this file is derived from.
• hzwer/Practical-RIFE — MIT; the README places the model weights under the same MIT license.
• Megvii ECCV2022-RIFE — MIT; the original Real-Time Intermediate Flow Estimation for Video Frame Interpolation.