video-benchmark-compact-quality

Compact Operator-Video Quality Model

Rate the quality of first-person / operator video (headband cameras, robotics teleop) — one tiny model, one forward pass, on-device.

Scoring a frame the "proper" way means running a whole stack of models: a learned image-quality ensemble, a CLIP scene classifier, a pile of OpenCV metrics. That's accurate, but it's hundreds of milliseconds and hundreds of megabytes per frame — you can't ship it to a headset or run it live.

So this model learns to imitate that stack. It watches the heavy pipeline label a big corpus of real operator footage, then compresses everything it learned into a single 2.5M-parameter network. You get all the same per-frame quality signals at once, on a plain 0–100 scale (higher is better) — fast enough for real time and small enough to embed.

At a glance

Try it in 30 seconds

import cv2
from huggingface_hub import hf_hub_download
# grab the model code (video_benchmark.distill) from https://github.com/shubhxho/video-benchmark
from video_benchmark.distill.infer import CompactQualityModel

ckpt = hf_hub_download("shubhxho/video-benchmark-compact-quality", "compact_quality.pt")
model = CompactQualityModel(ckpt)        # picks CUDA / Apple MPS / CPU automatically

frame = cv2.imread("frame.jpg")          # or any decoded video frame (BGR)
print(model.predict(frame))              # {'brightness': ..., 'iqa': ..., 'scene': ...}

Got a batch of frames? Call model.predict_batch([f1, f2, ...]) — it runs them together and, on a GPU, in fp16 for a free speedup.

What you get back

brightness, sharpness, blur, anomaly, iqa, musiq, clipiqa, scene

Two kinds of number live in there, and it's worth being honest about the difference:

• The learned signals (iqa, musiq, clipiqa, scene) are the whole point — perceptual quality and scene judgements that genuinely need a model. This is what the student reproduces.
• The rest are exact OpenCV statistics (brightness, sharpness, and friends). They're bundled so you get one tidy read-out, but if that's all you need, computing them directly is cheaper and exact.

How well does it copy the teacher?

• Throughput: 116.4× the teacher stack (1655 vs 14 fps)
• Composite PLCC: 0.899 (95% BCa CI [0.854, 0.925]) · fitted PLCC: 0.902 · KRCC: 0.714 · Deep PLCC: 0.685

Those are the standard IQA metrics — PLCC / SRCC / KRCC / MAE / RMSE — with a bootstrap 95% CI on the composite, all measured as student vs. teacher agreement. Want the full picture? Per-signal fidelity, training curves and a blur-robustness sweep are in report.md / report.txt, and the "why did we build it this way" story is in WRITEUP.md.

Under the hood

A frozen mobilenetv3_small_100 micro tower turns each frame into an embedding. That embedding gets fused with 14 cheap classical descriptors (luma, contrast, colourfulness, sharpness, edge density, exposure clipping) — the trick that lets the student recover absolute exposure/contrast, which an ImageNet tower otherwise normalises away. A small residual-MLP trunk refines the fused vector, and one tiny MLP head per signal reads off the final scores.

Where the scores come from

Per-signal fidelity is published in HF's eval-results format under .eval_results/results.yaml, keyed to the companion benchmark dataset shubhxho/operator-video-quality — one task per signal. These are self-reported distillation-fidelity numbers (how closely the student tracks the teacher); they roll up into a leaderboard once the benchmark is registered as a Hub Benchmark.

Graphs

Report dashboard

Calibration

Fidelity (PLCC / SRCC vs teacher)

fidelity_by_sigma.png

Residual spread

Robustness vs Gaussian blur

Student vs teacher — every signal

Throughput

Training loss & learning rate

On-device (candle / MLX)

Sub-fp16 weights for Rust / Apple-Silicon inference are bundled. Sizes are the real file bytes (container + weights), not a params×bytes estimate:

To keep int4 genuinely small, weights whose row length isn't a multiple of the quant block (many 1×1 conv / squeeze-excite kernels) are stored transposed so they block-quantise instead of falling back to fp16 — that alone trims the int4 build by ~20%. The transposed tensor names ride along in the metadata (compact.transposed for GGUF, transposed for MLX); a loader transposes them back after dequantising.

• candle (Rust): load the .gguf via candle_core::quantized::gguf_file (ggml Q4_0 / Q8_0). The trunk + heads are plain Linear / LayerNorm / GELU; GGUF metadata (compact.* keys) carries the arch, target order, the fused-descriptor names and the transposed-weight list so the forward pass — including the classical-descriptor fusion — is fully reconstructable.
• model.mxfp4.gguf (smallest): the same GGUF packed in MXFP4, the OCP Microscaling FP4 format frontier open models ship in — 4.25 bits/weight (vs int4's 4.5) for the smallest on-disk build, at a hair more quantisation error. Load it with a recent candle that supports MXFP4.
• MLX (Apple Silicon): load the .mlx.safetensors (grouped affine quant; arch + targets + descriptor names + transposed list in the safetensors metadata).

How it was trained

Self-distillation, end to end. The production pipeline at https://github.com/shubhxho/video-benchmark runs over an operator-video corpus and hands back per-frame teacher labels. Every sampled frame is then multiplied out with kornia Gaussian-blur + photometric augmentation — so signals that would otherwise be near-flat get enough spread to actually learn from — and the student regresses the teacher with SmoothL1 + a correlation loss, AdamW, SGDR cosine warm-restarts and an EMA of the weights.

Want one tuned to your footage? Point it at your own clips: python -m video_benchmark.distill.

License

The trained trunk + heads are MIT (this project). The bundled backbone weights derive from the timm mobilenetv3_small_100 ImageNet tower (Apache-2.0).