autoclip

AutoClip: ASD and face/person detection weights

This repository provides two existing checkpoints for AutoClip, a Python pipeline for full-length landscape-to-vertical video reframing with audio. These are upstream weights mirrored for convenience, not newly trained or fine-tuned models by shubhdotai.

Files

Both files live at the root of this Hugging Face repository. Download them into your AutoClip checkout's models/ directory without renaming them.

Download

With the Hugging Face CLI installed (pip install -U huggingface_hub):

hf download shubhdotai/autoclip pretrain_AVA.model yolov8x_person_face.pt --local-dir models

Or in Python:

from huggingface_hub import hf_hub_download

for filename in ("pretrain_AVA.model", "yolov8x_person_face.pt"):
    hf_hub_download("shubhdotai/autoclip", filename=filename, local_dir="models")

Within the AutoClip source checkout, its checksum-verifying downloader is:

python scripts/download_models.py                 # ASD only
python scripts/download_models.py --include-yolo  # both files

LR-ASD: active speaker detection

Source: Junhua-Liao/LR-ASD. The pretrain_AVA.model checkpoint was trained on AVA-ActiveSpeaker. It is a PyTorch state dictionary requiring the LR-ASD architecture and preprocessing; it is not a standalone Transformers model.

AutoClip supplies synchronized 112×112 grayscale face sequences and 13-value MFCC audio features, using 25 fps video and 16 kHz mono analysis audio. The network produces speaking scores per face track. The application uses a raw class-1 logit; this is not a calibrated probability. Audio alone is insufficient: the model compares audio with the visible face sequence.

From an installed AutoClip checkout, using macOS Vision for face detection:

autoclip run input.mp4 --output out/vertical

Apple Vision needs no separate detector weights. AutoClip selects MPS, CUDA or CPU for PyTorch according to device availability.

YOLOv8x: face and person detection

Source: iitolstykh/YOLO-Face-Person-Detector. This detector predicts face/person bounding boxes and confidence scores. AutoClip uses class 0 for person and class 1 for face. Detection alone does not determine who is speaking and does not maintain identities over time; tracking is performed separately by the application.

Use the YOLO detector in AutoClip:

autoclip run input.mp4 --detector yolo --output out/vertical-yolo

Or load it directly with Ultralytics:

from ultralytics import YOLO

model = YOLO("models/yolov8x_person_face.pt")
results = model.predict("frame.jpg", conf=0.4)
print(model.names)

The upstream card describes training on a proprietary face/person dataset. This mirror adds no training data or model changes. The original checkpoint can also be downloaded directly:

hf download iitolstykh/YOLO-Face-Person-Detector yolov8x_person_face.pt --local-dir models

Integrity and provenance

The ASD source is pinned to LR-ASD revision 1b6dcd2d8fc2895683de6508ec6294ec47d388ca, file weight/pretrain_AVA.model. The hashes identify the exact mirrored files. No upstream benchmark result is presented here as a new AutoClip evaluation.

Intended use and limitations

Intended for speaker-aware video framing with synchronized audio and visible faces. Small or occluded faces, profile views, dubbed audio, off-screen speech, crosstalk and rapid cuts can reduce quality. Neither checkpoint alone creates a reframed video; the application supplies tracking, scoring and rendering. The output retains the full video timeline, without subtitles or clip selection.

No new accuracy benchmark, demographic bias evaluation or 60-minute performance benchmark accompanies this mirror. Do not treat detection or speaker scores as reliable identity or other personal-attribute judgments.

License

Licenses apply per file. The MIT notice from LR-ASD applies to the ASD checkpoint; the YOLO upstream model card specifies AGPL-3.0. The ASD license does not relicense the YOLO checkpoint. Preserve upstream notices and refer to the LR-ASD license, the YOLO model license section, and the GNU AGPL-3.0 text. Ultralytics has its own licensing terms.

Citation

Credit the original model authors:

• Junhua Liao et al. A Light Weight Model for Active Speaker Detection. CVPR, 2023, pages 22932–22941.
• Junhua Liao et al. LR-ASD: Lightweight and Robust Network for Active Speaker Detection. International Journal of Computer Vision, 2025.
• For the face/person detector, use the attribution and citations provided in its upstream model card.