deepfake-detection

Each core returns the probability that the input is fake. The unified script combines them as a weighted sum:

final_fakeness = 0.5·lipsync + 0.4·audio + 0.1·video

(weights are renormalised over whichever cores actually produced a score).

---

How it routes input

        ┌──────────── audio file ───────────►  audio core  ──►  final = audio score
INPUT ──┤
        └──────────── video file ──┬─ extract wav ─► audio core  ┐
                                    ├──────────────► video core  ├─► weighted final
                                    └──────────────► lipsync core ┘

Audio-only input only the audio core runs; the final score is the audio score. Video input audio is extracted with ffmpeg and sent to the audio core; the video file goes to the video and lipsync cores.

Requirements

Linux ffmpeg / ffprobe** on PATH (the scripts default to /usr/bin/ffmpeg, /usr/bin/ffprobe). Three isolated Python environments, since the cores have mutually incompatible dependencies, so they cannot share one environment:

Put the environments on LOCAL disk (e.g. /opt)

Recreating the environments

# audio
python -m venv /opt/dfdetect-envs/audio
/opt/dfdetect-envs/audio/bin/python -m pip install -U pip wheel
/opt/dfdetect-envs/audio/bin/python -m pip install \
  torch==2.6.0 torchvision==0.21.0 transformers==4.44.0 \
  scikit-learn==1.3.2 scipy==1.13.1 joblib==1.4.2 \
  librosa soundfile accelerate "huggingface_hub<0.26" pandas numpy

# video
python -m venv /opt/dfdetect-envs/video
/opt/dfdetect-envs/video/bin/python -m pip install -U pip wheel
/opt/dfdetect-envs/video/bin/python -m pip install \
  torch==2.2.2 torchvision==0.17.2 transformers==4.51.2 numpy==1.26.4 \
  opencv-python==4.11.0.86 huggingface-hub safetensors accelerate pillow scipy pandas

# lipsync — a conda env (Python 3.10) named `avh`
conda create -n avh python=3.10 -y
/opt/conda/envs/avh/bin/python -m pip install -U pip wheel
/opt/conda/envs/avh/bin/python -m pip install -r avh-align_core/requirements.txt

The vendored avh-align_core/fairseq/ is the importable package only (no setup.py), so it works via sys.path at runtime but is not pip-installable — that is why the recipe above installs fairseq from source.

Inference — detect.py

python detect.py INPUT_FILE                 # text report
python detect.py INPUT_FILE --json          # machine-readable JSON

---

Retraining

Each script reuses the matching core's frozen feature extractor (so the feature space stays identical to inference) and retrains only the lightweight head. They auto-relaunch into the correct environment, so just run them with any Python.

Label conventions (important — audio is inverted!)

python retrain_audio.py --metadata audio_meta.csv

python retrain_video.py --metadata video_meta.csv --epochs 40

python retrain_lipsync.py --metadata real_videos.csv --epochs 10 --skip-existing

Each script has --help for all options (epochs, lr, batch size, --out, etc.).

Using a retrained model

Point detect.py at it — the weights are swapped in after the core loads:

python detect.py clip.mp4 \
  --audio-model   retrained_models/audio/logreg_retrained.joblib \
  --video-model   retrained_models/video/cnn_retrained.pt \
  --lipsync-model retrained_models/lipsync/fusion_retrained.pt

---

Troubleshooting

A core reports FAILED — its score is dropped and the remaining cores are reweighted; rerun with --verbose` to see that core's traceback (e.g. lipsync needs a detectable face; very short clips may yield no segments). No GPU / busy GPU — cores fall back to CPU (much slower). The box is shared, so throughput depends on how many of the GPUs are free.